-- Logs begin at Tue 2022-08-30 03:38:01 UTC, end at Tue 2022-08-30 04:48:11 UTC. -- Aug 30 04:11:44.870790 dv-cinder-854265-2-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Aug 30 04:11:44.879400 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Aug 30 04:11:44.882410 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: *** Starting uWSGI 2.0.18-debian (64bit) on [Tue Aug 30 04:11:44 2022] *** Aug 30 04:11:44.882410 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Aug 30 04:11:44.882410 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Aug 30 04:11:44.882410 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: nodename: dv-cinder-854265-2-iscsi Aug 30 04:11:44.882410 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: machine: x86_64 Aug 30 04:11:44.882410 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: clock source: unix Aug 30 04:11:44.882410 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: pcre jit disabled Aug 30 04:11:44.882410 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: detected number of CPU cores: 4 Aug 30 04:11:44.882410 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: current working directory: / Aug 30 04:11:44.882410 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: detected binary path: /usr/bin/uwsgi-core Aug 30 04:11:44.882410 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: your processes number limit is 47680 Aug 30 04:11:44.882410 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: your memory page size is 4096 bytes Aug 30 04:11:44.882410 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: detected max file descriptor number: 2048 Aug 30 04:11:44.883260 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: lock engine: pthread robust mutexes Aug 30 04:11:44.883260 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: thunder lock: enabled Aug 30 04:11:44.883260 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Aug 30 04:11:44.883260 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Aug 30 04:11:44.921892 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: Python main interpreter initialized at 0x55db356819c0 Aug 30 04:11:44.921892 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: python threads support enabled Aug 30 04:11:44.921892 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: your server socket listen backlog is limited to 100 connections Aug 30 04:11:44.921892 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: your mercy for graceful operations on workers is 90 seconds Aug 30 04:11:44.922297 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: mapped 403077 bytes (393 KB) for 2 cores Aug 30 04:11:44.922297 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: *** Operational MODE: preforking *** Aug 30 04:11:44.922582 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: *** uWSGI is running in multiple interpreter mode *** Aug 30 04:11:44.922582 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: spawned uWSGI master process (pid: 81468) Aug 30 04:11:44.922795 dv-cinder-854265-2-iscsi systemd[1]: Started Devstack devstack@keystone.service. Aug 30 04:11:44.924631 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: spawned uWSGI worker 1 (pid: 81484, cores: 1) Aug 30 04:11:44.924631 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: spawned uWSGI worker 2 (pid: 81485, cores: 1) Aug 30 04:11:44.924631 dv-cinder-854265-2-iscsi devstack@keystone.service[81468]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Aug 30 04:11:49.262091 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 30 04:11:49.262091 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: from cryptography.utils import int_from_bytes Aug 30 04:11:49.262899 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 30 04:11:49.262899 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: from cryptography.utils import int_from_bytes Aug 30 04:11:49.263020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 30 04:11:49.263020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: from cryptography.utils import int_from_bytes Aug 30 04:11:49.263020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 30 04:11:49.263020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: from cryptography.utils import int_from_bytes Aug 30 04:11:49.312239 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 30 04:11:49.312423 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 30 04:11:49.312503 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 30 04:11:49.312587 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 30 04:11:49.312664 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] command line args: [] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 30 04:11:49.312744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] command line args: [] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 30 04:11:49.312850 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 30 04:11:49.312959 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 30 04:11:49.313081 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ================================================================================ {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 30 04:11:49.313169 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ================================================================================ {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 30 04:11:49.313292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] admin_token = **** {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.313409 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] admin_token = **** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.313534 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] config_dir = [] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.313654 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] config_dir = [] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.313733 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.313840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.313918 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] config_source = [] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.314020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] config_source = [] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.314105 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] debug = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.314181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] debug = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.314252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.314524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.314657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.314762 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.314860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.314967 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.315069 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.315174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.315272 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.315373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.315477 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] list_limit = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.315573 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] list_limit = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.315674 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] log_config_append = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.315782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] log_config_append = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.315887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.315990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.316085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] log_dir = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.316186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] log_dir = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.316282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] log_file = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.316393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] log_file = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.316495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.316579 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.316704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.316785 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.316785 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.316905 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.317013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.317128 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.317203 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.317336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.317422 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.317541 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.317615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.317730 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.317840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.317931 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.318071 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.318193 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.318832 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.318946 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.319060 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.319141 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.319265 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.319347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.319492 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.319579 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.319735 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.319812 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.320005 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.320087 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.320486 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.85/identity {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.320568 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.85/identity {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.320647 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] publish_errors = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.320723 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] publish_errors = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.320879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.320964 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.321126 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.321215 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.321288 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.321413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.321529 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.321620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.321747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.321826 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.321976 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] standard_threads = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.322056 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] standard_threads = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.322185 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.322260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.322422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.322555 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.322631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.322699 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.322833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] use_journal = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.322907 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] use_journal = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.322982 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] use_json = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.323059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] use_json = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.323128 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] use_stderr = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.323197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] use_stderr = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.323277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] use_syslog = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.323348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] use_syslog = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.323417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.323486 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 04:11:49.323668 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.323757 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.323898 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.324242 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.324326 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.324404 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.324475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.324544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.324621 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.324693 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.324819 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.324897 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.325076 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.325171 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.325310 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.325388 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.325551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.325694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.325792 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.325876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.326032 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.326116 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.326260 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.326339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.326502 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.326579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.326730 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.326814 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.326973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.327061 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.327212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.327297 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.327445 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.327524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.327693 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.327775 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.327940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.328093 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.328166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.328234 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.328397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.328474 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.328644 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.328730 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.328903 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.328981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.329173 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.329252 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.329416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.329511 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.329643 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.329726 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.329867 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.329945 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.330092 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.330182 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.330320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.330398 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.330562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.330638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.330803 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.330890 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.331029 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.331106 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.331253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.331336 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.331493 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.331582 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.331733 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.331811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.331964 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.332048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.332205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.332290 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.332430 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.332512 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.332659 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.332743 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.332888 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.332974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.333138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.333217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.333363 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.333439 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.333589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.333742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.333820 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.333889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.334643 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.334860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.334944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.335066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.335129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.335211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.335300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.335494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.335583 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.335663 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.335740 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.335887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.335965 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.336116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.336201 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.336359 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.336467 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.336570 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.336668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.336807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.336922 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.337022 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.337179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.337277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.337423 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.337515 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.337600 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.337717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.337797 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.337942 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.338031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.338184 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.338318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.338396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.338514 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.338629 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.338757 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.338845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.338932 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.339099 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.339191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.339341 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.339458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.339567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.339706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.339782 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.339907 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.339990 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.340062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.340239 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.340321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.340473 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.340564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.340716 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.340816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.340934 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.341009 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.341190 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.341281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.341421 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.341505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.341653 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.341785 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.341909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.342043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.342168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.342302 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.342440 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.342569 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.342697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.342831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.342958 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.343085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.343205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.343329 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.343455 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.343588 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.343715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.343840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.343966 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.344091 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.344230 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.344364 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.344364 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.344596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.344722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.344849 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.344975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.345161 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.345293 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.345420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.345546 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.345672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.345798 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.345995 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.346134 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.346261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.346387 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.346513 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.346651 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.346786 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.346921 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.347049 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.347190 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.347325 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.347452 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.347578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.347710 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.347837 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.347976 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.348108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.348242 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.348370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.348503 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.348635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.348761 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.348887 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.349013 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.349163 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.349255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.349330 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.349407 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.349564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.349641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.349713 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.349813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.349895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] auth.external = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.349976 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] auth.external = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.350049 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.350119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.350196 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.350196 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.350332 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.350415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] auth.password = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.350487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.350565 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] auth.token = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.350638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] auth.password = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.350709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.350830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] auth.token = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.350916 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.350995 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.351067 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.351158 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.351230 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.351300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.351378 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.351457 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.351529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.351599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.351670 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.351745 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.351828 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] credential.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.351901 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.352063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.352143 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] credential.caching = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.352301 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.352379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.352545 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.352630 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.352787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.352865 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.353024 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.353161 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.353277 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.353357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.353504 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.353583 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.353745 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.353826 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.353984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.354064 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.354212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.354292 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.354459 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.354552 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.354736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.354816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.354977 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.355057 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.355223 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.355309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.355461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.355539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.355701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] federation.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.355779 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.355930 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] federation.caching = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.356016 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.356163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.356244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.356403 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.356486 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.356632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.356719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.356859 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.356938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.357128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.357251 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.357362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.357465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.357610 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.357688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.357842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.357928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.358085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.358172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.358316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.358395 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.358555 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.358633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.358783 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] identity.caching = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.358932 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.359030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.359116 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.359263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.359342 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.359493 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.359580 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.359722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.359803 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.359996 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.360082 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.360180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.360267 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.360413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.360492 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.360666 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.360770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.360897 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.361022 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.361150 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.361273 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.361380 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.361501 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.361622 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.361717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.361867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.361947 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.362112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.362191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.362366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.362453 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.362613 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.362692 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.362879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.362958 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.363117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.363201 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.363353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.363439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.363582 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.363662 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.363806 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.363886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.364036 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.364115 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.364260 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.364357 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.364492 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.364579 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.364726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.364805 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.364958 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.365066 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.365250 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.365331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.365477 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.365555 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.365719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.365798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.365953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.366033 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.366182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.366260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.366421 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.366501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.366649 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.366729 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.366876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.366954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.367102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.367179 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.367325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.367405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.367557 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.367637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.367789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.367869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.368024 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.368105 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.368258 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.368338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.368491 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.368570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.368734 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.368821 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.368963 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.369060 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.369217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.369296 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.369447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.369527 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.369690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.369777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.369921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.369999 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.370152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.370233 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.370382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.370463 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.370609 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.370688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.370832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.user = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.370912 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.371064 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.371143 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.371310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.371388 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.371535 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.371615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.371776 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.371855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.372014 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.372094 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.372245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.372324 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.372473 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.372552 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.372697 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.372776 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.372926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.373005 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.373174 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.373254 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.373402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.373479 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.373624 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.373710 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.373862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.373942 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.374101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.374180 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.374340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.374421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.374580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.374659 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.374806 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.374884 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.375031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.375110 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.375271 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.375350 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.375502 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.375582 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.375732 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.375813 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.375962 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.376039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.376189 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.376266 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.376428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.376509 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.376664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.376742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.376895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.376972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.377154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.377234 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.377390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.377468 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.377628 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.377717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.377857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.377935 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.378085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.378170 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.378315 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.378428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.378555 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.378672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.378879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.379351 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.379425 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.379496 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.379558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.379617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] resource.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.379674 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] resource.caching = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.379732 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.379789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.379855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.380019 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.380156 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.380282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.380407 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.380790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.381526 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.381657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.381787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.381916 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.382044 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.382171 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.382375 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.382507 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.382635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.382763 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.382890 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] role.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.383018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] role.caching = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.383153 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] role.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.383282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] role.driver = sql {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.383420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.383548 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.383675 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.383801 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.383936 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.384066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.384194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.384321 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.384448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.384576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.384711 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.384838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.384965 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.385121 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.385244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.385367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.385500 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.385625 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.385748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.385870 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.385992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.386114 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.386245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.386369 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.386492 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.386615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.386747 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.386871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.387002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.387126 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.387249 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.387372 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.387495 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.387627 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.387752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.387875 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.387998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.388120 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.388244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.388374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.388508 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.388633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.388756 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.388879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.389002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.389169 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.389292 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.389419 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.389545 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.389672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.389800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.389890 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.389965 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.390037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.390108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.390179 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.390250 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.390329 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.390403 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.390474 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.390546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.390617 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.390687 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.390765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.390838 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.390909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.390980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] token.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.391051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] token.caching = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.391121 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.391199 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.391272 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.391344 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.391414 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.391484 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.391580 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.391669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.391826 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.391906 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.392059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.392138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.392299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.392386 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.392535 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.392615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.392777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.392856 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.393011 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.393172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.393251 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.393322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.393486 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.393566 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.393726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.393813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.393997 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.394075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.394223 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.394311 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.394465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.394555 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.394726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.394806 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.394984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.connection = **** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.395064 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.connection = **** {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.395226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.395304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.395470 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.395550 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.395708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.395789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.395942 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.396021 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.396173 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.396261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.396425 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.396533 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.396715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.397386 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.397467 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.397631 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.397725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.397878 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.397958 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.398113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.398231 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.398354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.398442 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.398585 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.398665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.398827 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.398906 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.399064 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.399146 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.399290 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.399368 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.399519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.399596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.399748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.399873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.399951 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 30 04:11:49.400023 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 04:11:49.400128 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81484) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 30 04:11:49.429599 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.429827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.429979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.430115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.430196 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.430420 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.438431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.438776 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.466975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.467203 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.467365 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.467597 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.520888 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.522005 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.570231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.570460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.571119 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.571356 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.727410 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.727649 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.727878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.728203 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.728321 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.728405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.728480 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.728554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.728627 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.728836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.729068 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.729284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.751615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.751733 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.751830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.751910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.752056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.752141 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.752277 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.752360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.752492 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.752576 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.752705 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.752786 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.752916 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.753006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.764847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.764949 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.778323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.778558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.778815 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.778906 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.779042 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.779127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.779262 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.779347 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.779475 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.779560 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.779685 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.779769 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.779895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.780106 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81484) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 04:11:49.797006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.797136 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.797245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 30 04:11:49.797329 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 30 04:11:49.797593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.797677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.797794 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 30 04:11:49.797877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 30 04:11:49.798072 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.798154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.798337 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.798420 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.798602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.798843 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.798924 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.799105 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.809019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.809170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.809321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.809449 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.809593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.809721 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.809879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.810005 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.816831 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.817020 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81484) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.819770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.819934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81484) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.820058 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.820217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.820348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.820498 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.820633 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.820778 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.820919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.821076 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.821215 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.821345 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.821489 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.821614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.821767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.821887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.822041 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.822273 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.822355 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.822552 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.822632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.822820 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.845121 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.845272 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.846588 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.846734 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81484) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.846876 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.847019 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.851930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.852218 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.852320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81484) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.852502 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.852598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.852840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.858969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.859139 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81484) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.859268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.859423 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.863215 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.863512 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.863593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81484) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.863801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.863883 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.864087 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.864174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.864368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.864449 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.864723 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.864998 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.865348 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.865621 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.866443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.866736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.867527 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.883247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.883374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.883529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.883655 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.887411 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.887515 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81484) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.890541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.890702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.890843 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.890999 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.895185 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.895277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81484) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.898351 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81484) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.898435 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.901490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.901591 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.901793 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.901876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.902080 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.902163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.902365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.902448 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.902647 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.902729 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.902937 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.903026 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.919234 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81484) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.919335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.919521 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.919604 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.919788 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.919872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.920048 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.920131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.926829 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.926989 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.929061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.929253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.930621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.930760 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.930891 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.931017 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.933577 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.933730 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81484) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.933858 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.933982 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.934149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.934323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.934450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.934602 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.934718 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.934890 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.946013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.946116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81484) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.946302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.946384 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.946586 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.946695 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.946870 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.946978 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.947152 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.947262 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.947428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.947545 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.947735 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.947852 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.963091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.963189 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81484) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.966041 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81484) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.966167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.968901 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.969021 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.970349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.970499 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.971942 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81484) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.972031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.972233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.972315 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.972548 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.972651 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.979574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.979667 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.980991 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81484) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.981122 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.983931 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.984019 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.984236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.984316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.984526 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.984607 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.984812 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.984893 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.993299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.993397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81484) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:49.993595 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.993677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.993875 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:49.993954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.000697 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81484) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:50.000797 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 04:11:50.000983 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.001090 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.001296 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.001383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.001563 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.001657 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.001835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.001916 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.002112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.002193 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.002386 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.002477 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.002683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.002769 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.002943 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.003023 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.003219 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.003297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.003490 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.003611 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.003752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.003831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.004021 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.004153 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.004303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.004440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81484) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 04:11:50.038879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x55db356819c0 pid: 81485 (default app) Aug 30 04:11:50.039061 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x55db356819c0 pid: 81484 (default app) Aug 30 04:11:50.042390 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10480573-d3d8-4e25-a78a-40a48096b24d None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:11:50.042680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10480573-d3d8-4e25-a78a-40a48096b24d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:11:50.042952 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10480573-d3d8-4e25-a78a-40a48096b24d None None] PATH_INFO: `/v3` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:11:50.047126 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1/1] 10.222.0.85 () {50 vars in 791 bytes} [Tue Aug 30 04:11:50 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:11:56.604527 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e17b3e-7409-4e8a-99de-2d5f1b03d3fe None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:11:56.605107 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e17b3e-7409-4e8a-99de-2d5f1b03d3fe None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:11:56.605107 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e17b3e-7409-4e8a-99de-2d5f1b03d3fe None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:11:56.609328 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1/2] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:11:56 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:11:56.614609 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95f8a57-98bc-45b9-8798-869c2565d7ff None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:11:56.615111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95f8a57-98bc-45b9-8798-869c2565d7ff None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:11:56.615246 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95f8a57-98bc-45b9-8798-869c2565d7ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:11:56.687499 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: INFO dbcounter [None req-e95f8a57-98bc-45b9-8798-869c2565d7ff None None] Registered counter for database keystone Aug 30 04:11:56.706977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG oslo_db.sqlalchemy.engines [None req-e95f8a57-98bc-45b9-8798-869c2565d7ff None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81485) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 30 04:11:56.714397 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writer thread running {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 30 04:11:56.893965 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG passlib.handlers.bcrypt [None req-e95f8a57-98bc-45b9-8798-869c2565d7ff None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81485) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 30 04:11:56.894279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG passlib.handlers.bcrypt [None req-e95f8a57-98bc-45b9-8798-869c2565d7ff None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81485) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 30 04:11:56.953261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-e95f8a57-98bc-45b9-8798-869c2565d7ff None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:11:56.954849 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e95f8a57-98bc-45b9-8798-869c2565d7ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:11:57.081632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 2/3] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:11:56 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 468 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Aug 30 04:11:57.552499 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e17b3e-7409-4e8a-99de-2d5f1b03d3fe None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:11:57.632844 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: INFO dbcounter [None req-42e17b3e-7409-4e8a-99de-2d5f1b03d3fe None None] Registered counter for database keystone Aug 30 04:11:57.654517 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG oslo_db.sqlalchemy.engines [None req-42e17b3e-7409-4e8a-99de-2d5f1b03d3fe None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81484) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 30 04:11:57.660578 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writer thread running {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 30 04:11:57.778294 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e17b3e-7409-4e8a-99de-2d5f1b03d3fe None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:11:57.782467 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d38f22-6844-4d5f-bbbd-06563d6cc751 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:11:57.783093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d38f22-6844-4d5f-bbbd-06563d6cc751 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:11:57.783369 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d38f22-6844-4d5f-bbbd-06563d6cc751 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:11:57.783635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d38f22-6844-4d5f-bbbd-06563d6cc751 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:11:57.791638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d38f22-6844-4d5f-bbbd-06563d6cc751 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:11:57.843678 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: WARNING keystone.common.rbac_enforcer.enforcer [None req-f1d38f22-6844-4d5f-bbbd-06563d6cc751 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 30 04:11:57.843960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d38f22-6844-4d5f-bbbd-06563d6cc751 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:11:57.853512 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 2/4] 10.222.0.85 () {58 vars in 1324 bytes} [Tue Aug 30 04:11:57 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 04:11:57.859449 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95f8a57-98bc-45b9-8798-869c2565d7ff None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:11:57.874423 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95f8a57-98bc-45b9-8798-869c2565d7ff None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:11:57.878839 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb02ec7e-9712-453b-b684-e9a54d20e725 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:11:57.879949 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb02ec7e-9712-453b-b684-e9a54d20e725 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:11:57.880234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb02ec7e-9712-453b-b684-e9a54d20e725 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:11:57.880511 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb02ec7e-9712-453b-b684-e9a54d20e725 admin admin] PATH_INFO: `/v3/projects/be2e356bbe7d4680bd52a62e89c77545` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:11:57.882979 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb02ec7e-9712-453b-b684-e9a54d20e725 admin admin] RBAC: Authorizing `identity:get_project(project_id=be2e356bbe7d4680bd52a62e89c77545)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:11:57.935325 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: WARNING keystone.common.rbac_enforcer.enforcer [None req-bb02ec7e-9712-453b-b684-e9a54d20e725 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 30 04:11:57.935665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb02ec7e-9712-453b-b684-e9a54d20e725 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:11:57.940231 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 3/5] 10.222.0.85 () {56 vars in 1207 bytes} [Tue Aug 30 04:11:57 2022] GET /identity/v3/projects/be2e356bbe7d4680bd52a62e89c77545 => generated 344 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:11:59.824076 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f24b02b-fb59-4d79-8702-26eb5fbe43ff None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:11:59.824598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f24b02b-fb59-4d79-8702-26eb5fbe43ff None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:11:59.824703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f24b02b-fb59-4d79-8702-26eb5fbe43ff None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:11:59.829026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 3/6] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:11:59 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:11:59.834463 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac72514a-e753-4f91-ad76-0eff66bd1e32 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:11:59.834829 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac72514a-e753-4f91-ad76-0eff66bd1e32 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:11:59.835014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac72514a-e753-4f91-ad76-0eff66bd1e32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:11:59.855991 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ac72514a-e753-4f91-ad76-0eff66bd1e32 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:11:59.857561 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ac72514a-e753-4f91-ad76-0eff66bd1e32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:11:59.865984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 4/7] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:11:59 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Aug 30 04:12:00.143848 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f24b02b-fb59-4d79-8702-26eb5fbe43ff None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:00.156410 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f24b02b-fb59-4d79-8702-26eb5fbe43ff None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:00.160575 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d41c598-81a4-45f1-a25d-6e5209495776 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:00.161203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d41c598-81a4-45f1-a25d-6e5209495776 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:00.161486 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d41c598-81a4-45f1-a25d-6e5209495776 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:00.161758 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d41c598-81a4-45f1-a25d-6e5209495776 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:00.170056 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d41c598-81a4-45f1-a25d-6e5209495776 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:00.171837 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d41c598-81a4-45f1-a25d-6e5209495776 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:00.180413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 4/8] 10.222.0.85 () {58 vars in 1324 bytes} [Tue Aug 30 04:12:00 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 04:12:00.185914 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac72514a-e753-4f91-ad76-0eff66bd1e32 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:00.194990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac72514a-e753-4f91-ad76-0eff66bd1e32 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:00.199248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a07d1c-fbe6-4b32-9117-8e7e1fc0995d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:00.199861 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a07d1c-fbe6-4b32-9117-8e7e1fc0995d admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:00.200137 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a07d1c-fbe6-4b32-9117-8e7e1fc0995d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:00.200405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a07d1c-fbe6-4b32-9117-8e7e1fc0995d admin admin] PATH_INFO: `/v3/users/a85a3472aa4a4c95a2bc16b6450e8dcd` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:00.201721 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a07d1c-fbe6-4b32-9117-8e7e1fc0995d admin admin] RBAC: Authorizing `identity:get_user(user_id=a85a3472aa4a4c95a2bc16b6450e8dcd)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:00.203535 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a07d1c-fbe6-4b32-9117-8e7e1fc0995d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:00.205074 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 5/9] 10.222.0.85 () {56 vars in 1198 bytes} [Tue Aug 30 04:12:00 2022] GET /identity/v3/users/a85a3472aa4a4c95a2bc16b6450e8dcd => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:02.167233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cbfc3c-7fef-4831-8d77-443dc90ebcd1 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:02.167798 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cbfc3c-7fef-4831-8d77-443dc90ebcd1 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:02.167798 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cbfc3c-7fef-4831-8d77-443dc90ebcd1 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:02.169111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 5/10] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:02.174140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223055b6-2914-4ffd-a86d-edf9fc377a3e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:02.174569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223055b6-2914-4ffd-a86d-edf9fc377a3e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:02.174701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223055b6-2914-4ffd-a86d-edf9fc377a3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:02.193162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-223055b6-2914-4ffd-a86d-edf9fc377a3e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:02.195002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-223055b6-2914-4ffd-a86d-edf9fc377a3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:02.232917 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ce84e1-6a10-40ff-ab4d-5ea5d0b30ec1 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:02.233231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ce84e1-6a10-40ff-ab4d-5ea5d0b30ec1 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:02.233507 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ce84e1-6a10-40ff-ab4d-5ea5d0b30ec1 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:02.234835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 6/11] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:02.239758 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d321a1d-bcb0-4881-89cc-2ab2a5f1bfe3 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:02.240039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d321a1d-bcb0-4881-89cc-2ab2a5f1bfe3 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:02.240876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d321a1d-bcb0-4881-89cc-2ab2a5f1bfe3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:02.276911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 6/12] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:02.284008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG passlib.handlers.bcrypt [None req-8d321a1d-bcb0-4881-89cc-2ab2a5f1bfe3 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81484) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 30 04:12:02.284476 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG passlib.handlers.bcrypt [None req-8d321a1d-bcb0-4881-89cc-2ab2a5f1bfe3 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81484) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 30 04:12:02.334228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-8d321a1d-bcb0-4881-89cc-2ab2a5f1bfe3 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:02.335806 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8d321a1d-bcb0-4881-89cc-2ab2a5f1bfe3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:02.435513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 7/13] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:02.551373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223055b6-2914-4ffd-a86d-edf9fc377a3e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:02.575611 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:02.656080 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223055b6-2914-4ffd-a86d-edf9fc377a3e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:02.689852 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23bd137-0f45-4960-8078-994db19912db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:02.690417 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23bd137-0f45-4960-8078-994db19912db None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:02.690699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23bd137-0f45-4960-8078-994db19912db None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:02.690975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23bd137-0f45-4960-8078-994db19912db None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:02.700538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d321a1d-bcb0-4881-89cc-2ab2a5f1bfe3 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:02.704638 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23bd137-0f45-4960-8078-994db19912db None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:02.713709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-f23bd137-0f45-4960-8078-994db19912db None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:02.713709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:02.713709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:02.713709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:02.713709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:02.713709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:02.713709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:02.713709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:02.713709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:02.713709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:02.713709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:02.713709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:02.713709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:02.713709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:02.713709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:02.713709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:02.714708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:02.714708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:02.714708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:02.714708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:02.714708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:02.714708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:02.714708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:02.714708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:02.714708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:02.714708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:02.714708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:02.714708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:02.714708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:02.714708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:02.714708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:02.714708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:02.715632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:02.715632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:02.715632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:02.715632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:02.715632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:02.715632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:02.715632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:02.715632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:02.715632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:02.715632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:02.715632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:02.715632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:02.717174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 7/14] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:12:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:02.720643 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23bd137-0f45-4960-8078-994db19912db None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:02.805747 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:02.806102 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d321a1d-bcb0-4881-89cc-2ab2a5f1bfe3 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:02.808443 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:02.825430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23bd137-0f45-4960-8078-994db19912db None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:02.844940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24789fc5-dfda-4a0b-adec-3fcd7fd7f98a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:02.845832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24789fc5-dfda-4a0b-adec-3fcd7fd7f98a None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:02.846111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24789fc5-dfda-4a0b-adec-3fcd7fd7f98a None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:02.846400 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24789fc5-dfda-4a0b-adec-3fcd7fd7f98a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:02.859539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5d8d38-0329-4896-ac44-709d563d3eea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:02.860148 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5d8d38-0329-4896-ac44-709d563d3eea None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:02.860432 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5d8d38-0329-4896-ac44-709d563d3eea None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:02.860710 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5d8d38-0329-4896-ac44-709d563d3eea None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:02.861440 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5d8d38-0329-4896-ac44-709d563d3eea None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:02.861749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5d8d38-0329-4896-ac44-709d563d3eea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:02.863173 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5d8d38-0329-4896-ac44-709d563d3eea None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:02.874619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 8/15] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:02.880270 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5d8d38-0329-4896-ac44-709d563d3eea None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:02.914443 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24789fc5-dfda-4a0b-adec-3fcd7fd7f98a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:02.915961 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24789fc5-dfda-4a0b-adec-3fcd7fd7f98a None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:02.980732 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5d8d38-0329-4896-ac44-709d563d3eea None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:03.013845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 8/16] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:12:03.014881 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a80d0bb-792c-42ae-b59f-7eac7092b837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:03.015524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a80d0bb-792c-42ae-b59f-7eac7092b837 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:03.015805 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a80d0bb-792c-42ae-b59f-7eac7092b837 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:03.016081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a80d0bb-792c-42ae-b59f-7eac7092b837 None admin] PATH_INFO: `/v3/users/a85a3472aa4a4c95a2bc16b6450e8dcd` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:03.017266 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a80d0bb-792c-42ae-b59f-7eac7092b837 None admin] RBAC: Authorizing `identity:get_user(user_id=a85a3472aa4a4c95a2bc16b6450e8dcd)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:03.018738 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a80d0bb-792c-42ae-b59f-7eac7092b837 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:03.019593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24789fc5-dfda-4a0b-adec-3fcd7fd7f98a None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:03.020525 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 9/17] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:02 2022] GET /identity/v3/users/a85a3472aa4a4c95a2bc16b6450e8dcd => generated 247 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:03.025699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a80d0bb-792c-42ae-b59f-7eac7092b837 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:03.026724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:03.040440 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:03.123697 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24789fc5-dfda-4a0b-adec-3fcd7fd7f98a None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:03.133030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a80d0bb-792c-42ae-b59f-7eac7092b837 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:03.160227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ad75a9-9995-43f1-b625-ccabc35ff714 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:03.161100 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ad75a9-9995-43f1-b625-ccabc35ff714 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:03.161391 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ad75a9-9995-43f1-b625-ccabc35ff714 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:03.161656 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ad75a9-9995-43f1-b625-ccabc35ff714 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:03.169598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37d11ad-8bc8-476e-a8ed-5f562d834062 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:03.170259 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37d11ad-8bc8-476e-a8ed-5f562d834062 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:03.170542 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37d11ad-8bc8-476e-a8ed-5f562d834062 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:03.170817 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37d11ad-8bc8-476e-a8ed-5f562d834062 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:03.171933 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37d11ad-8bc8-476e-a8ed-5f562d834062 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:03.173415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37d11ad-8bc8-476e-a8ed-5f562d834062 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:03.174998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 10/18] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:12:03 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:03.180643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ad75a9-9995-43f1-b625-ccabc35ff714 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:03.181084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37d11ad-8bc8-476e-a8ed-5f562d834062 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:03.182143 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ad75a9-9995-43f1-b625-ccabc35ff714 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:03.200233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-07ad75a9-9995-43f1-b625-ccabc35ff714 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:12:03.200233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:03.200233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:03.200233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:03.200233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:03.200233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:03.200233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:03.200233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:03.200233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:03.200233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:03.200233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:03.200233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:03.200233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:12:03.200233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:12:03.200233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:12:03.200233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:12:03.201284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:03.201284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:03.201284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:03.201284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:03.201284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:03.201284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:03.201284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:03.201284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:03.201284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:03.201284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:03.201284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:03.201284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:03.201284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:03.201284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:03.201284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:03.201284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:03.201284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:12:03.202436 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:12:03.202436 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:12:03.202436 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:03.202436 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 9/19] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:12:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:12:03.207286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ad75a9-9995-43f1-b625-ccabc35ff714 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:03.263120 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:03.265113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:03.282514 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37d11ad-8bc8-476e-a8ed-5f562d834062 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:03.309282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ad75a9-9995-43f1-b625-ccabc35ff714 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:03.315517 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504a5e0a-370c-4d36-a51a-d96ab4412417 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:03.316134 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504a5e0a-370c-4d36-a51a-d96ab4412417 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:03.316413 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504a5e0a-370c-4d36-a51a-d96ab4412417 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:03.316687 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504a5e0a-370c-4d36-a51a-d96ab4412417 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:03.317454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504a5e0a-370c-4d36-a51a-d96ab4412417 None admin] RBAC: Adding query filter params (role.id=d17954e7ce544f0d90a512340232c60e, scope.domain.id=default, user.id=a85a3472aa4a4c95a2bc16b6450e8dcd) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:03.317840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504a5e0a-370c-4d36-a51a-d96ab4412417 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:03.319805 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504a5e0a-370c-4d36-a51a-d96ab4412417 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:03.326345 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 11/20] 10.222.0.85 () {56 vars in 1322 bytes} [Tue Aug 30 04:12:03 2022] GET /identity/v3/role_assignments?user.id=a85a3472aa4a4c95a2bc16b6450e8dcd&scope.domain.id=default&role.id=d17954e7ce544f0d90a512340232c60e => generated 233 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:03.344262 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09950228-54ad-48ee-a06e-5f67061b960e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:03.344904 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09950228-54ad-48ee-a06e-5f67061b960e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:03.345210 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09950228-54ad-48ee-a06e-5f67061b960e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:03.345482 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09950228-54ad-48ee-a06e-5f67061b960e None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:03.346143 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09950228-54ad-48ee-a06e-5f67061b960e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:03.346445 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09950228-54ad-48ee-a06e-5f67061b960e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:03.347876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09950228-54ad-48ee-a06e-5f67061b960e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:03.367256 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 10/21] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:12:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:05.541828 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322a0ae8-c3d1-4078-a8c1-b5d76a7832f5 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:05.542851 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322a0ae8-c3d1-4078-a8c1-b5d76a7832f5 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:05.543359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322a0ae8-c3d1-4078-a8c1-b5d76a7832f5 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:05.544980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 12/22] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:05.550460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851e9226-f927-46bd-b090-db9074e978d4 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:05.551533 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851e9226-f927-46bd-b090-db9074e978d4 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:05.552010 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851e9226-f927-46bd-b090-db9074e978d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:05.582067 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-851e9226-f927-46bd-b090-db9074e978d4 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:05.583653 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-851e9226-f927-46bd-b090-db9074e978d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:05.675871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 11/23] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:05.785830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bab960-9c26-4aaf-9735-9072f72946f6 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:05.786472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bab960-9c26-4aaf-9735-9072f72946f6 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:05.786976 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bab960-9c26-4aaf-9735-9072f72946f6 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:05.789634 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 13/24] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:05 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:05.795131 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c11f0e-b030-4126-8253-115913a84b91 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:05.795642 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c11f0e-b030-4126-8253-115913a84b91 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:05.796138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c11f0e-b030-4126-8253-115913a84b91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:05.817736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed49da0e-6424-4c92-9468-1fb2f07256ea None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:05.818427 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed49da0e-6424-4c92-9468-1fb2f07256ea None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:05.819008 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed49da0e-6424-4c92-9468-1fb2f07256ea None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:05.820948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 14/25] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:05.825556 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3167ae09-0c95-4c00-8197-83635bae388f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:05.826069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3167ae09-0c95-4c00-8197-83635bae388f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:05.826563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3167ae09-0c95-4c00-8197-83635bae388f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:05.830284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-23c11f0e-b030-4126-8253-115913a84b91 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:05.832561 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-23c11f0e-b030-4126-8253-115913a84b91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:05.862769 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3167ae09-0c95-4c00-8197-83635bae388f None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:05.864335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3167ae09-0c95-4c00-8197-83635bae388f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:05.938256 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:05.994643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 12/26] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:05.995787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b01f1e5-90d4-429b-9040-fb5367f18564 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:05.996141 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b01f1e5-90d4-429b-9040-fb5367f18564 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:05.996407 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b01f1e5-90d4-429b-9040-fb5367f18564 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:05.997675 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 13/27] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:06.004355 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446bfd49-2009-4746-9d38-3ce75ec139b2 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:06.004637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446bfd49-2009-4746-9d38-3ce75ec139b2 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:06.004909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446bfd49-2009-4746-9d38-3ce75ec139b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:06.033332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 15/28] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:06.044978 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-446bfd49-2009-4746-9d38-3ce75ec139b2 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:06.046514 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-446bfd49-2009-4746-9d38-3ce75ec139b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:06.074469 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3167ae09-0c95-4c00-8197-83635bae388f None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:06.169008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 14/29] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:06.239368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:06.276768 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3167ae09-0c95-4c00-8197-83635bae388f None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:06.322678 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc72896c-20bb-4970-870d-16ae5269e3c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:06.323249 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc72896c-20bb-4970-870d-16ae5269e3c3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:06.323530 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc72896c-20bb-4970-870d-16ae5269e3c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:06.323806 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc72896c-20bb-4970-870d-16ae5269e3c3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:06.336800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc72896c-20bb-4970-870d-16ae5269e3c3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:06.338324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc72896c-20bb-4970-870d-16ae5269e3c3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:06.358464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-fc72896c-20bb-4970-870d-16ae5269e3c3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:12:06.358464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:06.358464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:06.358464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:06.358464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:06.358464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:06.358464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:06.358464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:06.358464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:06.358464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:06.358464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:06.358464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:06.358464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:12:06.358464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:12:06.358464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:12:06.358464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:12:06.359322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:06.359322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:06.359322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:06.359322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:06.359322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:06.359322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:06.359322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:06.359322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:06.359322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:06.359322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:06.359322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:06.359322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:06.359322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:06.359322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:06.359322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:06.359322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:06.359322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:12:06.360119 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:12:06.360119 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:12:06.360119 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:06.360699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 16/30] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:12:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:06.365433 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446bfd49-2009-4746-9d38-3ce75ec139b2 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:06.369308 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc72896c-20bb-4970-870d-16ae5269e3c3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:06.505694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc72896c-20bb-4970-870d-16ae5269e3c3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:06.526855 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446bfd49-2009-4746-9d38-3ce75ec139b2 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:06.538599 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:06.545374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:06.549752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d937f2c1-971f-4fc7-8241-79b02242a5b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:06.550581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d937f2c1-971f-4fc7-8241-79b02242a5b7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:06.550901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d937f2c1-971f-4fc7-8241-79b02242a5b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:06.551170 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d937f2c1-971f-4fc7-8241-79b02242a5b7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:06.564047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d937f2c1-971f-4fc7-8241-79b02242a5b7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:06.569725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-d937f2c1-971f-4fc7-8241-79b02242a5b7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:06.569725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:06.569725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:06.569725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:06.569725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:06.569725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:06.569725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:06.569725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:06.569725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:06.569725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:06.569725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:06.569725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:06.569725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:06.569725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:06.569725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:06.569725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:06.570855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:06.570855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:06.570855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:06.570855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:06.570855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:06.570855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:06.570855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:06.570855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:06.570855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:06.570855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:06.570855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:06.570855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:06.570855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:06.570855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:06.570855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:06.570855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:06.571811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:06.571811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:06.571811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:06.571811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:06.571811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:06.571811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:06.571811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:06.571811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:06.571811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:06.571811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:06.571811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:06.571811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:06.573895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 17/31] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:12:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:06.574845 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d937f2c1-971f-4fc7-8241-79b02242a5b7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:06.588764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d03660-101b-4063-a260-2a76d082a45c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:06.589546 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d03660-101b-4063-a260-2a76d082a45c None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:06.589898 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d03660-101b-4063-a260-2a76d082a45c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:06.590241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d03660-101b-4063-a260-2a76d082a45c None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:06.590983 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d03660-101b-4063-a260-2a76d082a45c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:06.591355 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d03660-101b-4063-a260-2a76d082a45c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:06.593446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d03660-101b-4063-a260-2a76d082a45c None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:06.612934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 15/32] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:12:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:06.612934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d03660-101b-4063-a260-2a76d082a45c None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:06.717291 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d937f2c1-971f-4fc7-8241-79b02242a5b7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:06.737860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d03660-101b-4063-a260-2a76d082a45c None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:06.769003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1fdd71-d36d-4353-add8-29195168a4c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:06.770114 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1fdd71-d36d-4353-add8-29195168a4c1 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:06.770501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1fdd71-d36d-4353-add8-29195168a4c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:06.770883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1fdd71-d36d-4353-add8-29195168a4c1 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:06.772834 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1fdd71-d36d-4353-add8-29195168a4c1 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:06.774712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1fdd71-d36d-4353-add8-29195168a4c1 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:06.780801 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddd3982-81c8-41b0-a0c1-77eb45b71395 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:06.781604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddd3982-81c8-41b0-a0c1-77eb45b71395 None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:06.782444 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddd3982-81c8-41b0-a0c1-77eb45b71395 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:06.782915 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddd3982-81c8-41b0-a0c1-77eb45b71395 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:06.784070 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddd3982-81c8-41b0-a0c1-77eb45b71395 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:06.785815 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddd3982-81c8-41b0-a0c1-77eb45b71395 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:06.914901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 18/33] 10.222.0.85 () {60 vars in 1131 bytes} [Tue Aug 30 04:12:06 2022] POST /identity/v3/roles => generated 219 bytes in 340 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:12:06.916269 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1fdd71-d36d-4353-add8-29195168a4c1 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:06.916636 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 16/34] 10.222.0.85 () {60 vars in 1131 bytes} [Tue Aug 30 04:12:06 2022] POST /identity/v3/roles => generated 225 bytes in 307 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:12:06.917807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddd3982-81c8-41b0-a0c1-77eb45b71395 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:06.956951 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:06.994414 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:07.063818 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1fdd71-d36d-4353-add8-29195168a4c1 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:07.097570 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddd3982-81c8-41b0-a0c1-77eb45b71395 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:07.117872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b991d509-aee2-4918-9d3e-54d71033ad00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:07.119743 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b991d509-aee2-4918-9d3e-54d71033ad00 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:07.120039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b991d509-aee2-4918-9d3e-54d71033ad00 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:07.120624 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b991d509-aee2-4918-9d3e-54d71033ad00 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:07.122727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b991d509-aee2-4918-9d3e-54d71033ad00 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:07.123049 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b991d509-aee2-4918-9d3e-54d71033ad00 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:07.129068 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b991d509-aee2-4918-9d3e-54d71033ad00 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:07.136885 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 19/35] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:07.142739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b991d509-aee2-4918-9d3e-54d71033ad00 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:07.174063 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d35424-4e70-4793-bc68-2317dc67fdc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:07.174627 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d35424-4e70-4793-bc68-2317dc67fdc5 None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:07.174916 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d35424-4e70-4793-bc68-2317dc67fdc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:07.175197 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d35424-4e70-4793-bc68-2317dc67fdc5 None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:07.176235 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d35424-4e70-4793-bc68-2317dc67fdc5 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:07.178006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d35424-4e70-4793-bc68-2317dc67fdc5 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:07.225381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 17/36] 10.222.0.85 () {60 vars in 1141 bytes} [Tue Aug 30 04:12:06 2022] POST /identity/v3/projects => generated 301 bytes in 308 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:12:07.292456 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b991d509-aee2-4918-9d3e-54d71033ad00 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:07.304656 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:07.340012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36683c3a-4834-4c13-b536-833b846085f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:07.340679 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36683c3a-4834-4c13-b536-833b846085f8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:07.340966 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36683c3a-4834-4c13-b536-833b846085f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:07.341274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36683c3a-4834-4c13-b536-833b846085f8 None admin] PATH_INFO: `/v3/users/a85a3472aa4a4c95a2bc16b6450e8dcd` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:07.342552 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36683c3a-4834-4c13-b536-833b846085f8 None admin] RBAC: Authorizing `identity:get_user(user_id=a85a3472aa4a4c95a2bc16b6450e8dcd)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:07.344050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36683c3a-4834-4c13-b536-833b846085f8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:07.345921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 20/37] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:07 2022] GET /identity/v3/users/a85a3472aa4a4c95a2bc16b6450e8dcd => generated 247 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:07.351946 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d35424-4e70-4793-bc68-2317dc67fdc5 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:07.357436 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34b852e-9fb8-41be-93c5-f33d2693541c None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:07.358013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34b852e-9fb8-41be-93c5-f33d2693541c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:07.358459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34b852e-9fb8-41be-93c5-f33d2693541c None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:07.359988 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 21/38] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:12:07.370956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33eaf1fa-2050-4196-ae67-ed0461c6a09d None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:07.371630 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33eaf1fa-2050-4196-ae67-ed0461c6a09d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:07.372127 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33eaf1fa-2050-4196-ae67-ed0461c6a09d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:07.405138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-33eaf1fa-2050-4196-ae67-ed0461c6a09d None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:07.407302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-33eaf1fa-2050-4196-ae67-ed0461c6a09d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:07.453803 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:07.484102 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d35424-4e70-4793-bc68-2317dc67fdc5 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:07.518836 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 22/39] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:07.535746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0643fc53-4473-4d33-ab76-b464a3f8459e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:07.536394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0643fc53-4473-4d33-ab76-b464a3f8459e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:07.536679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0643fc53-4473-4d33-ab76-b464a3f8459e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:07.536954 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0643fc53-4473-4d33-ab76-b464a3f8459e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:07.538853 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0643fc53-4473-4d33-ab76-b464a3f8459e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:07.540328 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0643fc53-4473-4d33-ab76-b464a3f8459e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:07.546415 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 18/40] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:12:07 2022] GET /identity/v3/domains/default => generated 199 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:07.547897 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33eaf1fa-2050-4196-ae67-ed0461c6a09d None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:07.568294 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3742575-5403-4b35-a448-59a09d9804df None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:07.568579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3742575-5403-4b35-a448-59a09d9804df None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:07.568857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3742575-5403-4b35-a448-59a09d9804df None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:07.570237 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 19/41] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:12:07.575007 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed56619f-e649-4ceb-a640-01267cbc4612 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:07.575295 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed56619f-e649-4ceb-a640-01267cbc4612 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:07.575574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed56619f-e649-4ceb-a640-01267cbc4612 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:07.594150 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-ed56619f-e649-4ceb-a640-01267cbc4612 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:07.595667 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ed56619f-e649-4ceb-a640-01267cbc4612 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:07.663313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33eaf1fa-2050-4196-ae67-ed0461c6a09d None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:07.670780 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 20/42] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:07.777395 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:07.806681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128f8b9d-b399-45c0-a76f-fcb6798e54d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:07.807468 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128f8b9d-b399-45c0-a76f-fcb6798e54d6 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:07.807788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128f8b9d-b399-45c0-a76f-fcb6798e54d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:07.808358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128f8b9d-b399-45c0-a76f-fcb6798e54d6 None admin] PATH_INFO: `/v3/domains/default/users/a85a3472aa4a4c95a2bc16b6450e8dcd/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:07.810341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128f8b9d-b399-45c0-a76f-fcb6798e54d6 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=a85a3472aa4a4c95a2bc16b6450e8dcd, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:07.811829 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128f8b9d-b399-45c0-a76f-fcb6798e54d6 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:07.828348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 23/43] 10.222.0.85 () {58 vars in 1361 bytes} [Tue Aug 30 04:12:07 2022] PUT /identity/v3/domains/default/users/a85a3472aa4a4c95a2bc16b6450e8dcd/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:12:07.829439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed56619f-e649-4ceb-a640-01267cbc4612 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:07.926083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:07.942203 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128f8b9d-b399-45c0-a76f-fcb6798e54d6 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:07.956286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed56619f-e649-4ceb-a640-01267cbc4612 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:08.000784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66ef190-f414-4ffc-9a19-2df62c1faea3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:08.001460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66ef190-f414-4ffc-9a19-2df62c1faea3 None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:08.001728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66ef190-f414-4ffc-9a19-2df62c1faea3 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:08.002002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66ef190-f414-4ffc-9a19-2df62c1faea3 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:08.002679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d66ef190-f414-4ffc-9a19-2df62c1faea3 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:08.004087 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d66ef190-f414-4ffc-9a19-2df62c1faea3 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:08.019335 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 21/44] 10.222.0.85 () {60 vars in 1131 bytes} [Tue Aug 30 04:12:07 2022] POST /identity/v3/roles => generated 223 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:12:08.072886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128f8b9d-b399-45c0-a76f-fcb6798e54d6 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:08.120507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8935236c-e835-49e4-875c-0cc5acfb6bbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:08.121208 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8935236c-e835-49e4-875c-0cc5acfb6bbc None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:08.121541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8935236c-e835-49e4-875c-0cc5acfb6bbc None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:08.121824 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8935236c-e835-49e4-875c-0cc5acfb6bbc None admin] PATH_INFO: `/v3/domains/default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:08.123088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8935236c-e835-49e4-875c-0cc5acfb6bbc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:08.124590 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8935236c-e835-49e4-875c-0cc5acfb6bbc None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:08.126530 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 24/45] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:12:07 2022] GET /identity/v3/domains/default => generated 199 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:08.132446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66ef190-f414-4ffc-9a19-2df62c1faea3 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:08.247949 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66ef190-f414-4ffc-9a19-2df62c1faea3 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:08.281755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c4ec8a-32cb-49d0-bac0-38f815ed6d39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:08.282364 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c4ec8a-32cb-49d0-bac0-38f815ed6d39 None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:08.282644 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c4ec8a-32cb-49d0-bac0-38f815ed6d39 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:08.282938 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c4ec8a-32cb-49d0-bac0-38f815ed6d39 None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:08.283593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c4ec8a-32cb-49d0-bac0-38f815ed6d39 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:08.285024 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c4ec8a-32cb-49d0-bac0-38f815ed6d39 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:08.290533 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:08.312806 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 22/46] 10.222.0.85 () {60 vars in 1141 bytes} [Tue Aug 30 04:12:08 2022] POST /identity/v3/projects => generated 312 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:12:09.737965 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920a8aba-8b9a-464c-8cee-7a1c84b9d4ad None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:09.738512 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920a8aba-8b9a-464c-8cee-7a1c84b9d4ad None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:09.738512 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920a8aba-8b9a-464c-8cee-7a1c84b9d4ad None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:09.739873 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 25/47] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:09.745518 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09be6b5-add2-434a-b25a-59cd01b07486 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:09.745967 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09be6b5-add2-434a-b25a-59cd01b07486 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:09.746179 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09be6b5-add2-434a-b25a-59cd01b07486 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:09.766611 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-f09be6b5-add2-434a-b25a-59cd01b07486 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:09.768006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f09be6b5-add2-434a-b25a-59cd01b07486 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:09.843885 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 23/48] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:10.161949 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920a8aba-8b9a-464c-8cee-7a1c84b9d4ad None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:10.199182 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:10.203005 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91420da-d4f0-463a-8faa-889206c3c590 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:10.203345 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91420da-d4f0-463a-8faa-889206c3c590 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:10.203709 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91420da-d4f0-463a-8faa-889206c3c590 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:10.205080 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 24/49] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:12:10.210525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bd871b-95fa-41ff-aedb-3e159501573d None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:10.210812 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bd871b-95fa-41ff-aedb-3e159501573d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:10.211091 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bd871b-95fa-41ff-aedb-3e159501573d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:10.229328 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-e7bd871b-95fa-41ff-aedb-3e159501573d None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:10.230768 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e7bd871b-95fa-41ff-aedb-3e159501573d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:10.268449 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920a8aba-8b9a-464c-8cee-7a1c84b9d4ad None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:10.303298 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a07155-7d70-494d-90b9-033e4837e0da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:10.303927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a07155-7d70-494d-90b9-033e4837e0da None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:10.304208 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a07155-7d70-494d-90b9-033e4837e0da None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:10.304483 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a07155-7d70-494d-90b9-033e4837e0da None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:10.304605 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 25/50] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:10.314486 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a07155-7d70-494d-90b9-033e4837e0da None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:10.320035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-b7a07155-7d70-494d-90b9-033e4837e0da None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:10.320035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:10.320035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:10.320035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:10.320035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:10.320035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:10.320035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:10.320035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:10.320035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:10.320035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:10.320035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:10.320035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:10.320035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:10.320035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:10.320035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:10.320035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:10.320867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:10.320867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:10.320867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:10.320867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:10.320867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:10.320867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:10.320867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:10.320867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:10.320867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:10.320867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:10.320867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:10.320867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:10.320867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:10.320867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:10.320867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:10.320867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:10.322147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:10.322147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:10.322147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:10.322147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:10.322147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:10.322147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:10.322147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:10.322147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:10.322147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:10.322147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:10.322147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:10.322147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:10.322147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 26/51] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:12:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:10.327109 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bd871b-95fa-41ff-aedb-3e159501573d None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:10.375179 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:10.425433 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bd871b-95fa-41ff-aedb-3e159501573d None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:10.458994 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0936e46-12fe-44d1-b9d8-260610e17380 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:10.459598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0936e46-12fe-44d1-b9d8-260610e17380 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:10.459881 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0936e46-12fe-44d1-b9d8-260610e17380 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:10.460154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0936e46-12fe-44d1-b9d8-260610e17380 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:10.460813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0936e46-12fe-44d1-b9d8-260610e17380 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:10.461147 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0936e46-12fe-44d1-b9d8-260610e17380 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:10.462527 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0936e46-12fe-44d1-b9d8-260610e17380 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:10.475461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 26/52] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:10.481603 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a07155-7d70-494d-90b9-033e4837e0da None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:10.571231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0936e46-12fe-44d1-b9d8-260610e17380 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:10.573526 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:10.580497 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a07155-7d70-494d-90b9-033e4837e0da None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:10.614360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce849a3-b1f2-4729-9cc2-ec15fea56c3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:10.615019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce849a3-b1f2-4729-9cc2-ec15fea56c3d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:10.615841 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce849a3-b1f2-4729-9cc2-ec15fea56c3d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:10.616122 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce849a3-b1f2-4729-9cc2-ec15fea56c3d None admin] PATH_INFO: `/v3/users/a85a3472aa4a4c95a2bc16b6450e8dcd` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:10.617352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce849a3-b1f2-4729-9cc2-ec15fea56c3d None admin] RBAC: Authorizing `identity:get_user(user_id=a85a3472aa4a4c95a2bc16b6450e8dcd)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:10.618848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce849a3-b1f2-4729-9cc2-ec15fea56c3d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:10.620600 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 27/53] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:10 2022] GET /identity/v3/users/a85a3472aa4a4c95a2bc16b6450e8dcd => generated 247 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:10.626890 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce849a3-b1f2-4729-9cc2-ec15fea56c3d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:10.735077 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce849a3-b1f2-4729-9cc2-ec15fea56c3d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:10.771071 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6496d5-a1b5-4e02-9989-67840667ca71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:10.771724 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6496d5-a1b5-4e02-9989-67840667ca71 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:10.772008 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6496d5-a1b5-4e02-9989-67840667ca71 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:10.772323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6496d5-a1b5-4e02-9989-67840667ca71 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:10.773655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6496d5-a1b5-4e02-9989-67840667ca71 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:10.775131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6496d5-a1b5-4e02-9989-67840667ca71 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:10.776704 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 28/54] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:12:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:10.779127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0936e46-12fe-44d1-b9d8-260610e17380 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:10.782462 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6496d5-a1b5-4e02-9989-67840667ca71 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:10.790315 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:10.812917 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:10.815333 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1f0087-72f8-4083-92ea-a520c3f8799c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:10.815980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1f0087-72f8-4083-92ea-a520c3f8799c None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:10.816273 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1f0087-72f8-4083-92ea-a520c3f8799c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:10.816538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1f0087-72f8-4083-92ea-a520c3f8799c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:10.817652 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1f0087-72f8-4083-92ea-a520c3f8799c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:10.819646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1f0087-72f8-4083-92ea-a520c3f8799c None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:10.821255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 27/55] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:12:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:10.826685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1f0087-72f8-4083-92ea-a520c3f8799c None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:10.889463 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6496d5-a1b5-4e02-9989-67840667ca71 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:10.928623 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd1bebc-4dde-4804-ac14-9f96b73c3651 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:10.928812 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1f0087-72f8-4083-92ea-a520c3f8799c None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:10.929265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd1bebc-4dde-4804-ac14-9f96b73c3651 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:10.929549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd1bebc-4dde-4804-ac14-9f96b73c3651 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:10.929821 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd1bebc-4dde-4804-ac14-9f96b73c3651 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:10.930545 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd1bebc-4dde-4804-ac14-9f96b73c3651 None admin] RBAC: Adding query filter params (role.id=d17954e7ce544f0d90a512340232c60e, scope.domain.id=default, user.id=a85a3472aa4a4c95a2bc16b6450e8dcd) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:10.930853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd1bebc-4dde-4804-ac14-9f96b73c3651 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:10.932309 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd1bebc-4dde-4804-ac14-9f96b73c3651 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:10.938110 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 29/56] 10.222.0.85 () {56 vars in 1322 bytes} [Tue Aug 30 04:12:10 2022] GET /identity/v3/role_assignments?user.id=a85a3472aa4a4c95a2bc16b6450e8dcd&scope.domain.id=default&role.id=d17954e7ce544f0d90a512340232c60e => generated 530 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:10.965483 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acfe091-bb3e-429d-bb3a-0580c47712cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:10.966116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acfe091-bb3e-429d-bb3a-0580c47712cf None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:10.966401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acfe091-bb3e-429d-bb3a-0580c47712cf None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:10.966677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acfe091-bb3e-429d-bb3a-0580c47712cf None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:10.967365 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7acfe091-bb3e-429d-bb3a-0580c47712cf None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:10.968860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7acfe091-bb3e-429d-bb3a-0580c47712cf None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:10.999378 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 28/57] 10.222.0.85 () {60 vars in 1140 bytes} [Tue Aug 30 04:12:10 2022] POST /identity/v3/projects => generated 298 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:12:12.902355 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b0ef78-b8cc-4f32-9484-12d560213a2a None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:12.902893 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b0ef78-b8cc-4f32-9484-12d560213a2a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:12.902893 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b0ef78-b8cc-4f32-9484-12d560213a2a None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:12.904246 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 30/58] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:12.909520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968e00dd-f604-45e4-8e51-078e282a1d78 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:12.909914 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968e00dd-f604-45e4-8e51-078e282a1d78 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:12.910161 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968e00dd-f604-45e4-8e51-078e282a1d78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:12.928139 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-968e00dd-f604-45e4-8e51-078e282a1d78 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:12.929359 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-968e00dd-f604-45e4-8e51-078e282a1d78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:12.952516 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:12.996011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 29/59] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:13.263332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b0ef78-b8cc-4f32-9484-12d560213a2a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:13.354045 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:13.361370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b0ef78-b8cc-4f32-9484-12d560213a2a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:13.394040 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9a39a3-8171-4f3a-b315-d22f5fec9a91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:13.394670 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9a39a3-8171-4f3a-b315-d22f5fec9a91 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:13.394951 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9a39a3-8171-4f3a-b315-d22f5fec9a91 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:13.395228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9a39a3-8171-4f3a-b315-d22f5fec9a91 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:13.396351 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9a39a3-8171-4f3a-b315-d22f5fec9a91 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:13.398408 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9a39a3-8171-4f3a-b315-d22f5fec9a91 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:13.399964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 31/60] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:12:13 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:13.405396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968e00dd-f604-45e4-8e51-078e282a1d78 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:13.499848 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968e00dd-f604-45e4-8e51-078e282a1d78 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:13.532163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648eb325-79a8-4257-bdee-cb869ecefc58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:13.532700 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648eb325-79a8-4257-bdee-cb869ecefc58 None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:13.532976 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648eb325-79a8-4257-bdee-cb869ecefc58 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:13.533272 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648eb325-79a8-4257-bdee-cb869ecefc58 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:13.533924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648eb325-79a8-4257-bdee-cb869ecefc58 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:13.535359 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648eb325-79a8-4257-bdee-cb869ecefc58 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:13.562165 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 30/61] 10.222.0.85 () {60 vars in 1132 bytes} [Tue Aug 30 04:12:13 2022] POST /identity/v3/users => generated 275 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:12:16.652015 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739d4f89-916d-46f9-8775-fdfece3432b9 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:16.652866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739d4f89-916d-46f9-8775-fdfece3432b9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:16.653303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739d4f89-916d-46f9-8775-fdfece3432b9 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:16.654788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 32/62] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:16.659908 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9e87e2-da98-4d28-a9f1-eb7e9d8f0f82 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:16.660735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9e87e2-da98-4d28-a9f1-eb7e9d8f0f82 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:16.661173 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9e87e2-da98-4d28-a9f1-eb7e9d8f0f82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:16.686009 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-9f9e87e2-da98-4d28-a9f1-eb7e9d8f0f82 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:16.687422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-9f9e87e2-da98-4d28-a9f1-eb7e9d8f0f82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:16.703502 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:16.785479 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 31/63] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:16.802522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d33461-1a86-4af9-b502-60527fcf69c4 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:16.803039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d33461-1a86-4af9-b502-60527fcf69c4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:16.803455 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d33461-1a86-4af9-b502-60527fcf69c4 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:16.804962 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 33/64] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:16.810080 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c5e810-c563-4311-9bbe-315347cb84c3 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:16.810525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c5e810-c563-4311-9bbe-315347cb84c3 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:16.810942 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c5e810-c563-4311-9bbe-315347cb84c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:16.832895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba1aac5-f5be-4641-b19b-79b79c42f4cb None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:16.833450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba1aac5-f5be-4641-b19b-79b79c42f4cb None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:16.833893 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba1aac5-f5be-4641-b19b-79b79c42f4cb None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:16.835941 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 34/65] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:12:16.840087 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-f3c5e810-c563-4311-9bbe-315347cb84c3 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:16.841463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f3c5e810-c563-4311-9bbe-315347cb84c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:16.850112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48da16dc-c447-4c3a-9b4a-d64aae6f8c0d None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:16.850618 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48da16dc-c447-4c3a-9b4a-d64aae6f8c0d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:16.851035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48da16dc-c447-4c3a-9b4a-d64aae6f8c0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:16.875274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-48da16dc-c447-4c3a-9b4a-d64aae6f8c0d None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:16.876699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-48da16dc-c447-4c3a-9b4a-d64aae6f8c0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:16.965125 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 32/66] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:16.979174 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4dd604-de03-48cc-99a4-76c4c8da6511 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:16.979457 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4dd604-de03-48cc-99a4-76c4c8da6511 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:16.979734 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4dd604-de03-48cc-99a4-76c4c8da6511 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:16.981214 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 33/67] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:16.993317 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fb9c73-b2b7-442a-8751-96ee2931ba64 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:16.994380 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fb9c73-b2b7-442a-8751-96ee2931ba64 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:16.994852 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fb9c73-b2b7-442a-8751-96ee2931ba64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:17.002389 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 35/68] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:17.018351 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-80fb9c73-b2b7-442a-8751-96ee2931ba64 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:17.019812 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-80fb9c73-b2b7-442a-8751-96ee2931ba64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:17.106064 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48da16dc-c447-4c3a-9b4a-d64aae6f8c0d None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:17.121968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 34/69] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:17.200051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8032f21-68b6-4cea-a704-7c7a55785e31 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:17.200357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8032f21-68b6-4cea-a704-7c7a55785e31 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:17.200602 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8032f21-68b6-4cea-a704-7c7a55785e31 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:17.201975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 35/70] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:17.212544 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44af8854-1f51-4e5f-88fa-0f9710e0fe05 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:17.213219 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44af8854-1f51-4e5f-88fa-0f9710e0fe05 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:17.213713 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44af8854-1f51-4e5f-88fa-0f9710e0fe05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:17.233199 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:17.239006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:17.246343 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-44af8854-1f51-4e5f-88fa-0f9710e0fe05 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:17.249084 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-44af8854-1f51-4e5f-88fa-0f9710e0fe05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:17.258065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48da16dc-c447-4c3a-9b4a-d64aae6f8c0d None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:17.320705 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4d0a50-cfff-4ca2-8b24-7d4d88d0cf26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:17.321404 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4d0a50-cfff-4ca2-8b24-7d4d88d0cf26 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:17.321690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4d0a50-cfff-4ca2-8b24-7d4d88d0cf26 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:17.321972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4d0a50-cfff-4ca2-8b24-7d4d88d0cf26 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:17.334731 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4d0a50-cfff-4ca2-8b24-7d4d88d0cf26 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:17.341416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-aa4d0a50-cfff-4ca2-8b24-7d4d88d0cf26 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:17.341416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:17.341416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:17.341416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:17.341416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:17.341416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:17.341416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:17.341416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:17.341416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:17.341416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:17.341416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:17.341416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:17.341416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:17.341416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:17.341416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:17.341416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:17.342278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:17.342278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:17.342278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:17.342278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:17.342278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:17.342278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:17.342278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:17.342278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:17.342278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:17.342278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:17.342278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:17.342278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:17.342278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:17.342278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:17.342278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:17.342278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:17.343039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:17.343039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:17.343039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:17.343039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:17.343039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:17.343039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:17.343039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:17.343039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:17.343039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:17.343039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:17.343039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:17.343039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:17.344225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 36/71] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:12:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:12:17.345089 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4d0a50-cfff-4ca2-8b24-7d4d88d0cf26 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:17.357247 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 36/72] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:17.357247 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44af8854-1f51-4e5f-88fa-0f9710e0fe05 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:17.459221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44af8854-1f51-4e5f-88fa-0f9710e0fe05 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:17.459403 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4d0a50-cfff-4ca2-8b24-7d4d88d0cf26 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:17.495048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766c8681-a77a-4b2b-81c4-0ccdd5e52367 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:17.495230 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1383366-6c3c-42da-9f67-aee79bd2cc2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:17.495614 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766c8681-a77a-4b2b-81c4-0ccdd5e52367 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:17.495784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1383366-6c3c-42da-9f67-aee79bd2cc2b None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:17.495931 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766c8681-a77a-4b2b-81c4-0ccdd5e52367 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:17.496080 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1383366-6c3c-42da-9f67-aee79bd2cc2b None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:17.496163 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766c8681-a77a-4b2b-81c4-0ccdd5e52367 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:17.496323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1383366-6c3c-42da-9f67-aee79bd2cc2b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:17.499891 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:17.507593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766c8681-a77a-4b2b-81c4-0ccdd5e52367 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:17.509856 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1383366-6c3c-42da-9f67-aee79bd2cc2b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:17.512482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-766c8681-a77a-4b2b-81c4-0ccdd5e52367 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:17.512482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:17.512482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:17.512482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:17.512482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:17.512482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:17.512482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:17.512482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:17.512482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:17.512482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:17.512482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:17.512482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:17.512482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:17.512482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:17.512482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:17.512482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:17.513491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:17.513491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:17.513491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:17.513491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:17.513491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:17.513491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:17.513491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:17.513491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:17.513491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:17.513491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:17.513491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:17.513491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:17.513491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:17.513491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:17.513491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:17.513491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:17.514503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:17.514503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:17.514503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:17.514503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:17.514503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:17.514503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:17.514503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:17.514503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:17.514503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:17.514503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:17.514503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:17.514503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:17.514503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 37/73] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:17.516664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766c8681-a77a-4b2b-81c4-0ccdd5e52367 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:17.516764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-f1383366-6c3c-42da-9f67-aee79bd2cc2b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:12:17.516764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:17.516764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:17.516764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:17.516764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:17.516764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:17.516764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:17.516764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:17.516764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:17.516764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:17.516764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:17.516764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:17.516764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:17.516764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:17.516764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:17.516764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:17.517580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:17.517580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:17.517580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:17.517580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:17.517580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:17.517580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:17.517580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:17.517580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:17.517580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:17.517580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:17.517580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:17.517580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:17.517580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:17.517580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:17.517580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:17.517580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:17.518838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:17.518838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:17.518838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:17.518838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:17.518838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:17.518838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:17.518838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:17.518838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:17.518838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:17.518838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:17.518838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:12:17.518838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:17.518838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 37/74] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:12:17 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:17.518838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1383366-6c3c-42da-9f67-aee79bd2cc2b None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:17.538779 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:17.626491 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1383366-6c3c-42da-9f67-aee79bd2cc2b None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:17.631327 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766c8681-a77a-4b2b-81c4-0ccdd5e52367 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:17.661448 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d469da45-2025-4033-ab31-39525f719d50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:17.662681 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d469da45-2025-4033-ab31-39525f719d50 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:17.663043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d469da45-2025-4033-ab31-39525f719d50 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:17.663370 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d469da45-2025-4033-ab31-39525f719d50 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:17.664651 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d469da45-2025-4033-ab31-39525f719d50 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:17.665663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3999042-d426-4345-938e-7f71ebbe33fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:17.666276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3999042-d426-4345-938e-7f71ebbe33fa None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:17.666559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3999042-d426-4345-938e-7f71ebbe33fa None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:17.666664 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d469da45-2025-4033-ab31-39525f719d50 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:17.666831 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3999042-d426-4345-938e-7f71ebbe33fa None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:17.667523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3999042-d426-4345-938e-7f71ebbe33fa None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:17.667827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3999042-d426-4345-938e-7f71ebbe33fa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:17.668455 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 38/75] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:12:17 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:17.669240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3999042-d426-4345-938e-7f71ebbe33fa None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:17.669844 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d469da45-2025-4033-ab31-39525f719d50 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:17.677524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 38/76] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:17.678915 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3999042-d426-4345-938e-7f71ebbe33fa None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:17.734308 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:17.753683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:17.775152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d469da45-2025-4033-ab31-39525f719d50 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:17.780501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3999042-d426-4345-938e-7f71ebbe33fa None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:17.808380 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea06ba9-2cdb-4a8c-bb05-421210609835 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:17.808985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea06ba9-2cdb-4a8c-bb05-421210609835 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:17.809291 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea06ba9-2cdb-4a8c-bb05-421210609835 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:17.809571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea06ba9-2cdb-4a8c-bb05-421210609835 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:17.810227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea06ba9-2cdb-4a8c-bb05-421210609835 None admin] RBAC: Adding query filter params (name=member) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:17.810561 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea06ba9-2cdb-4a8c-bb05-421210609835 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:17.811939 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea06ba9-2cdb-4a8c-bb05-421210609835 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:17.813618 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48bcadf-bb2a-4b02-91f6-6640d4f44378 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:17.814310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48bcadf-bb2a-4b02-91f6-6640d4f44378 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:17.814659 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48bcadf-bb2a-4b02-91f6-6640d4f44378 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:17.814944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48bcadf-bb2a-4b02-91f6-6640d4f44378 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:17.815622 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48bcadf-bb2a-4b02-91f6-6640d4f44378 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:17.815931 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48bcadf-bb2a-4b02-91f6-6640d4f44378 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:17.817337 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48bcadf-bb2a-4b02-91f6-6640d4f44378 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:17.820282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 39/77] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:17 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:17.821098 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea06ba9-2cdb-4a8c-bb05-421210609835 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:17.828366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 39/78] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:12:17 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:17.829200 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48bcadf-bb2a-4b02-91f6-6640d4f44378 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:17.922580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea06ba9-2cdb-4a8c-bb05-421210609835 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:17.928764 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48bcadf-bb2a-4b02-91f6-6640d4f44378 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:17.947081 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:17.956556 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b689a13f-47ec-44a3-b92a-89f2f710b91a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:17.957237 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b689a13f-47ec-44a3-b92a-89f2f710b91a None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:17.957524 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b689a13f-47ec-44a3-b92a-89f2f710b91a None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:17.957799 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b689a13f-47ec-44a3-b92a-89f2f710b91a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:17.962591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7cb7a2-5609-4bd5-b84d-aa9039dc5f2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:17.963353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7cb7a2-5609-4bd5-b84d-aa9039dc5f2f None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:17.963635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7cb7a2-5609-4bd5-b84d-aa9039dc5f2f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:17.963911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7cb7a2-5609-4bd5-b84d-aa9039dc5f2f None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:17.964561 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7cb7a2-5609-4bd5-b84d-aa9039dc5f2f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:17.966034 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7cb7a2-5609-4bd5-b84d-aa9039dc5f2f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:17.968347 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b689a13f-47ec-44a3-b92a-89f2f710b91a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:17.973679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-b689a13f-47ec-44a3-b92a-89f2f710b91a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:17.973679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:17.973679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:17.973679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:17.973679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:17.973679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:17.973679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:17.973679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:17.973679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:17.973679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:17.973679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:17.973679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:17.973679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:17.973679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:17.973679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:17.973679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:17.974558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:17.974558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:17.974558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:17.974558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:17.974558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:17.974558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:17.974558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:17.974558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:17.974558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:17.974558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:17.974558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:17.974558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:17.974558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:17.974558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:17.974558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:17.974558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:17.975861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:17.975861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:17.975861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:17.975861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:17.975861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:17.975861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:17.975861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:17.975861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:17.975861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:17.975861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:17.975861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:17.975861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:17.975861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 40/79] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:17.976472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b689a13f-47ec-44a3-b92a-89f2f710b91a None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:17.980864 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:17.995609 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 40/80] 10.222.0.85 () {60 vars in 1132 bytes} [Tue Aug 30 04:12:17 2022] POST /identity/v3/users => generated 289 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:12:18.001339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7cb7a2-5609-4bd5-b84d-aa9039dc5f2f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:18.098108 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b689a13f-47ec-44a3-b92a-89f2f710b91a None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:18.131263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7cb7a2-5609-4bd5-b84d-aa9039dc5f2f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:18.139501 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd26859-37ad-42b5-844a-7b9d44747500 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:18.140147 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd26859-37ad-42b5-844a-7b9d44747500 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:18.140428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd26859-37ad-42b5-844a-7b9d44747500 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:18.140694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd26859-37ad-42b5-844a-7b9d44747500 None admin] PATH_INFO: `/v3/users/a85a3472aa4a4c95a2bc16b6450e8dcd` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:18.142165 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd26859-37ad-42b5-844a-7b9d44747500 None admin] RBAC: Authorizing `identity:get_user(user_id=a85a3472aa4a4c95a2bc16b6450e8dcd)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:18.143612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd26859-37ad-42b5-844a-7b9d44747500 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:18.145449 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 41/81] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:17 2022] GET /identity/v3/users/a85a3472aa4a4c95a2bc16b6450e8dcd => generated 247 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:18.146010 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd26859-37ad-42b5-844a-7b9d44747500 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:18.177316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2af76bb-2664-471f-a54a-a81b49268a74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:18.178803 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2af76bb-2664-471f-a54a-a81b49268a74 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:18.179145 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2af76bb-2664-471f-a54a-a81b49268a74 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:18.179487 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2af76bb-2664-471f-a54a-a81b49268a74 None admin] PATH_INFO: `/v3/users/6a1eb60c46aa4d27b91657cd7f9fa60f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:18.195996 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2af76bb-2664-471f-a54a-a81b49268a74 None admin] RBAC: Authorizing `identity:get_user(user_id=6a1eb60c46aa4d27b91657cd7f9fa60f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:18.197522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2af76bb-2664-471f-a54a-a81b49268a74 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:18.199563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 41/82] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:17 2022] GET /identity/v3/users/6a1eb60c46aa4d27b91657cd7f9fa60f => generated 275 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:18.200022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2af76bb-2664-471f-a54a-a81b49268a74 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:18.214604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:18.271211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:18.276709 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd26859-37ad-42b5-844a-7b9d44747500 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:18.317801 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83239014-5d5f-4f48-9f5f-a973b879b037 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:18.318551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83239014-5d5f-4f48-9f5f-a973b879b037 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:18.318840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83239014-5d5f-4f48-9f5f-a973b879b037 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:18.319111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83239014-5d5f-4f48-9f5f-a973b879b037 None admin] PATH_INFO: `/v3/users/6a1eb60c46aa4d27b91657cd7f9fa60f` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:18.320277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83239014-5d5f-4f48-9f5f-a973b879b037 None admin] RBAC: Authorizing `identity:get_user(user_id=6a1eb60c46aa4d27b91657cd7f9fa60f)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:18.321768 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83239014-5d5f-4f48-9f5f-a973b879b037 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:18.323381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 42/83] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:18 2022] GET /identity/v3/users/6a1eb60c46aa4d27b91657cd7f9fa60f => generated 275 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:18.328278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83239014-5d5f-4f48-9f5f-a973b879b037 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:18.331461 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2af76bb-2664-471f-a54a-a81b49268a74 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:18.374441 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77dcf4cf-cf1d-479a-9c18-0633ffb034d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:18.374988 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dcf4cf-cf1d-479a-9c18-0633ffb034d5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:18.375276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dcf4cf-cf1d-479a-9c18-0633ffb034d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:18.375556 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dcf4cf-cf1d-479a-9c18-0633ffb034d5 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:18.376241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77dcf4cf-cf1d-479a-9c18-0633ffb034d5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:18.376549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77dcf4cf-cf1d-479a-9c18-0633ffb034d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:18.378111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77dcf4cf-cf1d-479a-9c18-0633ffb034d5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:18.387516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 42/84] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:18.388420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77dcf4cf-cf1d-479a-9c18-0633ffb034d5 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:18.434059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83239014-5d5f-4f48-9f5f-a973b879b037 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:18.457590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:18.470040 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e561f18-c111-4514-b4ff-ed9e8ca8a5ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:18.470660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e561f18-c111-4514-b4ff-ed9e8ca8a5ed None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:18.470942 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e561f18-c111-4514-b4ff-ed9e8ca8a5ed None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:18.471216 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e561f18-c111-4514-b4ff-ed9e8ca8a5ed None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:18.498587 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77dcf4cf-cf1d-479a-9c18-0633ffb034d5 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:18.512872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:18.537532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b7996b-22a7-4f92-ba23-b2b38c65fbbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:18.538423 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b7996b-22a7-4f92-ba23-b2b38c65fbbe None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:18.538878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b7996b-22a7-4f92-ba23-b2b38c65fbbe None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:18.539295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b7996b-22a7-4f92-ba23-b2b38c65fbbe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:18.543799 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e561f18-c111-4514-b4ff-ed9e8ca8a5ed None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:18.545024 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e561f18-c111-4514-b4ff-ed9e8ca8a5ed None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:18.623143 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b7996b-22a7-4f92-ba23-b2b38c65fbbe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:18.625529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b7996b-22a7-4f92-ba23-b2b38c65fbbe None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:18.661052 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 43/85] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:12:18.666554 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e561f18-c111-4514-b4ff-ed9e8ca8a5ed None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:18.696007 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:18.764005 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 43/86] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:12:18.766216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b7996b-22a7-4f92-ba23-b2b38c65fbbe None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:18.782846 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e561f18-c111-4514-b4ff-ed9e8ca8a5ed None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:18.786800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:18.826401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33f1d37-f770-466a-8307-7b786316ddfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:18.827111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33f1d37-f770-466a-8307-7b786316ddfb None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:18.827394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33f1d37-f770-466a-8307-7b786316ddfb None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:18.827664 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33f1d37-f770-466a-8307-7b786316ddfb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:18.902089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33f1d37-f770-466a-8307-7b786316ddfb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:18.903341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b7996b-22a7-4f92-ba23-b2b38c65fbbe None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:18.903470 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33f1d37-f770-466a-8307-7b786316ddfb None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:18.929062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:18.953398 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6759e23-2556-40e5-a7b7-8c95a99208ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:18.954038 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6759e23-2556-40e5-a7b7-8c95a99208ff None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:18.954595 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6759e23-2556-40e5-a7b7-8c95a99208ff None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:18.954595 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6759e23-2556-40e5-a7b7-8c95a99208ff None admin] PATH_INFO: `/v3/users/6a1eb60c46aa4d27b91657cd7f9fa60f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:18.955749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6759e23-2556-40e5-a7b7-8c95a99208ff None admin] RBAC: Authorizing `identity:get_user(user_id=6a1eb60c46aa4d27b91657cd7f9fa60f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:18.957380 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6759e23-2556-40e5-a7b7-8c95a99208ff None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:18.959558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 44/87] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:18 2022] GET /identity/v3/users/6a1eb60c46aa4d27b91657cd7f9fa60f => generated 275 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:18.960399 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6759e23-2556-40e5-a7b7-8c95a99208ff None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:19.028633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 44/88] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:12:19.029839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33f1d37-f770-466a-8307-7b786316ddfb None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:19.048836 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:19.072051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6759e23-2556-40e5-a7b7-8c95a99208ff None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:19.116075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05899e3-4e5f-4158-b4e8-c55f3ce519b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:19.117147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05899e3-4e5f-4158-b4e8-c55f3ce519b4 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:19.117634 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05899e3-4e5f-4158-b4e8-c55f3ce519b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:19.118108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05899e3-4e5f-4158-b4e8-c55f3ce519b4 None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:19.122421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05899e3-4e5f-4158-b4e8-c55f3ce519b4 None admin] RBAC: Authorizing `identity:get_project(project_id=26661cf5060b4b939d8ab02f9ea45daf)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:19.124242 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05899e3-4e5f-4158-b4e8-c55f3ce519b4 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:19.126391 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 45/89] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:18 2022] GET /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:19.127359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05899e3-4e5f-4158-b4e8-c55f3ce519b4 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:19.141297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33f1d37-f770-466a-8307-7b786316ddfb None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:19.166627 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:19.180529 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777e45bf-d020-41c8-b77e-744380583db9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:19.181258 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777e45bf-d020-41c8-b77e-744380583db9 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:19.181539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777e45bf-d020-41c8-b77e-744380583db9 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:19.181810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777e45bf-d020-41c8-b77e-744380583db9 None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:19.183181 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777e45bf-d020-41c8-b77e-744380583db9 None admin] RBAC: Authorizing `identity:get_project(project_id=26661cf5060b4b939d8ab02f9ea45daf)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:19.184622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777e45bf-d020-41c8-b77e-744380583db9 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:19.186338 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 45/90] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:19 2022] GET /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf => generated 298 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:19.187182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777e45bf-d020-41c8-b77e-744380583db9 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:19.245353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05899e3-4e5f-4158-b4e8-c55f3ce519b4 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:19.283863 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be291f1d-7a99-4c75-8e38-a089baa0eeca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:19.284972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be291f1d-7a99-4c75-8e38-a089baa0eeca None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:19.285511 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be291f1d-7a99-4c75-8e38-a089baa0eeca None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:19.286000 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be291f1d-7a99-4c75-8e38-a089baa0eeca None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:19.295173 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777e45bf-d020-41c8-b77e-744380583db9 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:19.308140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:19.334547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad73089-cbf7-41d7-936c-d263abb0858e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:19.335193 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad73089-cbf7-41d7-936c-d263abb0858e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:19.335471 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad73089-cbf7-41d7-936c-d263abb0858e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:19.335740 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad73089-cbf7-41d7-936c-d263abb0858e None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:19.336971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad73089-cbf7-41d7-936c-d263abb0858e None admin] RBAC: Authorizing `identity:get_project(project_id=26661cf5060b4b939d8ab02f9ea45daf)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:19.338625 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad73089-cbf7-41d7-936c-d263abb0858e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:19.340590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 46/91] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:19 2022] GET /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf => generated 298 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:19.341263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad73089-cbf7-41d7-936c-d263abb0858e None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:19.366897 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be291f1d-7a99-4c75-8e38-a089baa0eeca None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:19.368850 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be291f1d-7a99-4c75-8e38-a089baa0eeca None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:19.404399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:19.450644 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad73089-cbf7-41d7-936c-d263abb0858e None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:19.478587 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 46/92] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:12:19.479849 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be291f1d-7a99-4c75-8e38-a089baa0eeca None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:19.488533 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464799db-a763-4eb9-8ef9-2c654729c71e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:19.489184 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464799db-a763-4eb9-8ef9-2c654729c71e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:19.489471 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464799db-a763-4eb9-8ef9-2c654729c71e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:19.489760 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464799db-a763-4eb9-8ef9-2c654729c71e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:19.490606 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464799db-a763-4eb9-8ef9-2c654729c71e None admin] RBAC: Adding query filter params (role.id=d17954e7ce544f0d90a512340232c60e, scope.project.id=26661cf5060b4b939d8ab02f9ea45daf, user.id=a85a3472aa4a4c95a2bc16b6450e8dcd) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:19.490911 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464799db-a763-4eb9-8ef9-2c654729c71e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:19.492425 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464799db-a763-4eb9-8ef9-2c654729c71e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:19.501618 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 47/93] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:12:19 2022] GET /identity/v3/role_assignments?user.id=a85a3472aa4a4c95a2bc16b6450e8dcd&scope.project.id=26661cf5060b4b939d8ab02f9ea45daf&role.id=d17954e7ce544f0d90a512340232c60e => generated 259 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:19.507720 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464799db-a763-4eb9-8ef9-2c654729c71e None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:19.536307 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:19.628495 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be291f1d-7a99-4c75-8e38-a089baa0eeca None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:19.642942 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464799db-a763-4eb9-8ef9-2c654729c71e None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:19.668652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97ada20-cfb2-432e-8443-a57ee6f27687 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:19.669341 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:19.670894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97ada20-cfb2-432e-8443-a57ee6f27687 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:19.671412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97ada20-cfb2-432e-8443-a57ee6f27687 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:19.671937 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97ada20-cfb2-432e-8443-a57ee6f27687 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:19.673648 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97ada20-cfb2-432e-8443-a57ee6f27687 None admin] RBAC: Adding query filter params (role.id=ae0cef8120cf4dfc899c30803243a851, scope.project.id=26661cf5060b4b939d8ab02f9ea45daf, user.id=6a1eb60c46aa4d27b91657cd7f9fa60f) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:19.674246 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97ada20-cfb2-432e-8443-a57ee6f27687 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:19.676186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97ada20-cfb2-432e-8443-a57ee6f27687 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:19.690451 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c70e594-451c-470e-86cb-b83af621d532 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:19.691090 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c70e594-451c-470e-86cb-b83af621d532 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:19.692061 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c70e594-451c-470e-86cb-b83af621d532 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:19.692061 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c70e594-451c-470e-86cb-b83af621d532 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:19.692705 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c70e594-451c-470e-86cb-b83af621d532 None admin] RBAC: Adding query filter params (role.id=a9e862ac528245ca8363bccc5bf35287, scope.project.id=26661cf5060b4b939d8ab02f9ea45daf, user.id=6a1eb60c46aa4d27b91657cd7f9fa60f) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:19.693015 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c70e594-451c-470e-86cb-b83af621d532 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:19.696361 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c70e594-451c-470e-86cb-b83af621d532 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:19.698323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 47/94] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:12:19 2022] GET /identity/v3/role_assignments?user.id=6a1eb60c46aa4d27b91657cd7f9fa60f&scope.project.id=26661cf5060b4b939d8ab02f9ea45daf&role.id=ae0cef8120cf4dfc899c30803243a851 => generated 259 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:19.699192 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97ada20-cfb2-432e-8443-a57ee6f27687 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:19.715923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 48/95] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:12:19 2022] GET /identity/v3/role_assignments?user.id=6a1eb60c46aa4d27b91657cd7f9fa60f&scope.project.id=26661cf5060b4b939d8ab02f9ea45daf&role.id=a9e862ac528245ca8363bccc5bf35287 => generated 259 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:19.847041 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97ada20-cfb2-432e-8443-a57ee6f27687 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:19.863592 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:19.889019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a3465f-00cc-4bfb-8e77-609b3ae07825 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:19.890181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a3465f-00cc-4bfb-8e77-609b3ae07825 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:19.890672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a3465f-00cc-4bfb-8e77-609b3ae07825 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:19.891153 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a3465f-00cc-4bfb-8e77-609b3ae07825 None admin] PATH_INFO: `/v3/projects/c20f0dd259eb4d55856b1f0848964837` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:19.892680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a3465f-00cc-4bfb-8e77-609b3ae07825 None admin] RBAC: Authorizing `identity:get_project(project_id=c20f0dd259eb4d55856b1f0848964837)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:19.898683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a3465f-00cc-4bfb-8e77-609b3ae07825 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:19.900874 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 48/96] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:19 2022] GET /identity/v3/projects/c20f0dd259eb4d55856b1f0848964837 => generated 312 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:19.907080 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c70e594-451c-470e-86cb-b83af621d532 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:20.047401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c70e594-451c-470e-86cb-b83af621d532 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:20.088067 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a4d6ba-0382-4d24-b94c-096e5cc119ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:20.088700 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a4d6ba-0382-4d24-b94c-096e5cc119ca None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:20.088983 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a4d6ba-0382-4d24-b94c-096e5cc119ca None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:20.089284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a4d6ba-0382-4d24-b94c-096e5cc119ca None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:20.090025 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a4d6ba-0382-4d24-b94c-096e5cc119ca None admin] RBAC: Adding query filter params (role.id=ae0cef8120cf4dfc899c30803243a851, scope.project.id=c20f0dd259eb4d55856b1f0848964837, user.id=6a1eb60c46aa4d27b91657cd7f9fa60f) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:20.090339 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a4d6ba-0382-4d24-b94c-096e5cc119ca None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:20.091912 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a4d6ba-0382-4d24-b94c-096e5cc119ca None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:20.098574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 49/97] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:12:19 2022] GET /identity/v3/role_assignments?user.id=6a1eb60c46aa4d27b91657cd7f9fa60f&scope.project.id=c20f0dd259eb4d55856b1f0848964837&role.id=ae0cef8120cf4dfc899c30803243a851 => generated 259 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:20.244360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0435ef3b-3117-4a80-84a4-abe8efde01fc None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:20.244674 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0435ef3b-3117-4a80-84a4-abe8efde01fc None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:20.244923 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0435ef3b-3117-4a80-84a4-abe8efde01fc None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:20.246335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 49/98] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:20.251862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba95b533-327e-4d42-8c82-96e52a8afa48 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:20.252152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba95b533-327e-4d42-8c82-96e52a8afa48 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:20.252429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba95b533-327e-4d42-8c82-96e52a8afa48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:20.276981 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-ba95b533-327e-4d42-8c82-96e52a8afa48 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:20.278217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ba95b533-327e-4d42-8c82-96e52a8afa48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:20.345105 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:20.378119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 50/99] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:20.382484 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a44c9e-0f2d-4150-823f-75793dab3511 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:20.382774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a44c9e-0f2d-4150-823f-75793dab3511 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:20.383056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a44c9e-0f2d-4150-823f-75793dab3511 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:20.384401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 50/100] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:20.401028 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc64a9f9-2d9c-40b9-8530-cd8704f89062 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:20.401333 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc64a9f9-2d9c-40b9-8530-cd8704f89062 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:20.401615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc64a9f9-2d9c-40b9-8530-cd8704f89062 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:20.431115 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-cc64a9f9-2d9c-40b9-8530-cd8704f89062 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:20.432307 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-cc64a9f9-2d9c-40b9-8530-cd8704f89062 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:20.527421 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 51/101] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:20.827769 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a44c9e-0f2d-4150-823f-75793dab3511 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:20.901587 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc64a9f9-2d9c-40b9-8530-cd8704f89062 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:20.978261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a44c9e-0f2d-4150-823f-75793dab3511 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:21.010691 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc64a9f9-2d9c-40b9-8530-cd8704f89062 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:21.023373 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:21.053958 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6870af-92e5-437d-b965-9bd1c4c20d53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:21.056770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6870af-92e5-437d-b965-9bd1c4c20d53 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:21.057454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6870af-92e5-437d-b965-9bd1c4c20d53 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:21.058555 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6870af-92e5-437d-b965-9bd1c4c20d53 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:21.071468 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9a406d-fe81-403e-b34c-5e9db53a7455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:21.072423 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9a406d-fe81-403e-b34c-5e9db53a7455 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:21.073902 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9a406d-fe81-403e-b34c-5e9db53a7455 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:21.074374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9a406d-fe81-403e-b34c-5e9db53a7455 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:21.076781 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9a406d-fe81-403e-b34c-5e9db53a7455 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:21.081013 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9a406d-fe81-403e-b34c-5e9db53a7455 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:21.082898 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 52/102] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:12:20 2022] GET /identity/v3/domains/default => generated 199 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:21.088969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6870af-92e5-437d-b965-9bd1c4c20d53 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:21.090941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9a406d-fe81-403e-b34c-5e9db53a7455 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:21.098559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-9c6870af-92e5-437d-b965-9bd1c4c20d53 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:12:21.098559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:21.098559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:21.098559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:21.098559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:21.098559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:21.098559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:21.098559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:21.098559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:21.098559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:21.098559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:21.098559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:21.098559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:21.098559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:21.098559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:21.098559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:21.099465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:21.099465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:21.099465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:21.099465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:21.099465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:21.099465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:21.099465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:21.099465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:21.099465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:21.099465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:21.099465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:21.099465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:21.099465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:21.099465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:21.099465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:21.099465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:21.100263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:21.100263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:21.100263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:21.100263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:21.100263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:21.100263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:21.100263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:21.100263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:21.100263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:21.100263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:21.100263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:12:21.100263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:21.102992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 51/103] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:20 2022] GET /identity/v3/roles/reader => generated 84 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:21.108277 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6870af-92e5-437d-b965-9bd1c4c20d53 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:21.176390 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:21.198165 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9a406d-fe81-403e-b34c-5e9db53a7455 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:21.240816 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908d5722-da5e-40d3-9e24-1a4109e8e527 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:21.241560 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908d5722-da5e-40d3-9e24-1a4109e8e527 None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:21.241847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908d5722-da5e-40d3-9e24-1a4109e8e527 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:21.242121 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908d5722-da5e-40d3-9e24-1a4109e8e527 None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:21.243008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908d5722-da5e-40d3-9e24-1a4109e8e527 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:21.244556 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908d5722-da5e-40d3-9e24-1a4109e8e527 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:21.265227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6870af-92e5-437d-b965-9bd1c4c20d53 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:21.289197 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 53/104] 10.222.0.85 () {60 vars in 1141 bytes} [Tue Aug 30 04:12:21 2022] POST /identity/v3/projects => generated 302 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:12:21.322325 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67d87de-18d3-48d7-9d17-5ec331c602de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:21.322973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67d87de-18d3-48d7-9d17-5ec331c602de None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:21.323258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67d87de-18d3-48d7-9d17-5ec331c602de None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:21.323538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67d87de-18d3-48d7-9d17-5ec331c602de None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:21.324226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e67d87de-18d3-48d7-9d17-5ec331c602de None admin] RBAC: Adding query filter params (name=reader) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:21.324532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e67d87de-18d3-48d7-9d17-5ec331c602de None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:21.326084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e67d87de-18d3-48d7-9d17-5ec331c602de None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:21.340987 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 52/105] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:21 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:21.346980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908d5722-da5e-40d3-9e24-1a4109e8e527 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:21.392994 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:21.474158 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908d5722-da5e-40d3-9e24-1a4109e8e527 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:21.511716 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840aae8b-31fb-44e8-907e-b8d0eb90e66a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:21.512362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840aae8b-31fb-44e8-907e-b8d0eb90e66a None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:21.512643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840aae8b-31fb-44e8-907e-b8d0eb90e66a None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:21.512916 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840aae8b-31fb-44e8-907e-b8d0eb90e66a None admin] PATH_INFO: `/v3/users/3124ffc09dfd4a52880634dbc9fcad43` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:21.528635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840aae8b-31fb-44e8-907e-b8d0eb90e66a None admin] RBAC: Authorizing `identity:get_user(user_id=3124ffc09dfd4a52880634dbc9fcad43)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:21.530134 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840aae8b-31fb-44e8-907e-b8d0eb90e66a None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:21.532107 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 54/106] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:21 2022] GET /identity/v3/users/3124ffc09dfd4a52880634dbc9fcad43 => generated 289 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:21.538223 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67d87de-18d3-48d7-9d17-5ec331c602de None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:21.651499 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67d87de-18d3-48d7-9d17-5ec331c602de None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:21.652964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:21.696673 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a218c0-bab6-4577-9dcc-9d0ef185bd17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:21.697333 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a218c0-bab6-4577-9dcc-9d0ef185bd17 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:21.697614 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a218c0-bab6-4577-9dcc-9d0ef185bd17 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:21.697888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a218c0-bab6-4577-9dcc-9d0ef185bd17 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:21.788076 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a218c0-bab6-4577-9dcc-9d0ef185bd17 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:21.789598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a218c0-bab6-4577-9dcc-9d0ef185bd17 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:21.909703 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 53/107] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:12:21.915718 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840aae8b-31fb-44e8-907e-b8d0eb90e66a None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:22.036046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:22.053081 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840aae8b-31fb-44e8-907e-b8d0eb90e66a None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:22.089568 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad41f87-d232-44a5-9881-446598a50110 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:22.090565 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad41f87-d232-44a5-9881-446598a50110 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:22.091052 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad41f87-d232-44a5-9881-446598a50110 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:22.091527 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad41f87-d232-44a5-9881-446598a50110 None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:22.092867 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad41f87-d232-44a5-9881-446598a50110 None admin] RBAC: Authorizing `identity:get_project(project_id=26661cf5060b4b939d8ab02f9ea45daf)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:22.094606 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad41f87-d232-44a5-9881-446598a50110 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:22.096465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 55/108] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:21 2022] GET /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf => generated 298 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:22.103952 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a218c0-bab6-4577-9dcc-9d0ef185bd17 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:22.115447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:22.243421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a218c0-bab6-4577-9dcc-9d0ef185bd17 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:22.290694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3121c44c-fc16-44a4-867f-002fc5fd0dc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:22.291668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3121c44c-fc16-44a4-867f-002fc5fd0dc1 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:22.292150 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3121c44c-fc16-44a4-867f-002fc5fd0dc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:22.292627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3121c44c-fc16-44a4-867f-002fc5fd0dc1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:22.293621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3121c44c-fc16-44a4-867f-002fc5fd0dc1 None admin] RBAC: Adding query filter params (role.id=902fd8d9a4ef48058e1b17e0429ede58, scope.project.id=26661cf5060b4b939d8ab02f9ea45daf, user.id=3124ffc09dfd4a52880634dbc9fcad43) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:22.294132 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3121c44c-fc16-44a4-867f-002fc5fd0dc1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:22.295772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3121c44c-fc16-44a4-867f-002fc5fd0dc1 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:22.305402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 54/109] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:12:22 2022] GET /identity/v3/role_assignments?user.id=3124ffc09dfd4a52880634dbc9fcad43&scope.project.id=26661cf5060b4b939d8ab02f9ea45daf&role.id=902fd8d9a4ef48058e1b17e0429ede58 => generated 259 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:22.675097 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85a1470-eb4c-4c8a-b48f-aa7884a799bd None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:22.675759 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85a1470-eb4c-4c8a-b48f-aa7884a799bd None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:22.676248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85a1470-eb4c-4c8a-b48f-aa7884a799bd None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:22.678378 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 56/110] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:22.688958 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1928f2fe-cf9c-4d4d-bf8d-743c32c672e3 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:22.689560 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1928f2fe-cf9c-4d4d-bf8d-743c32c672e3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:22.690036 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1928f2fe-cf9c-4d4d-bf8d-743c32c672e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:22.707080 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4577711a-4a88-4fc6-9c56-c76db0dd9c2b None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:22.707674 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4577711a-4a88-4fc6-9c56-c76db0dd9c2b None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:22.708172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4577711a-4a88-4fc6-9c56-c76db0dd9c2b None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:22.709787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 57/111] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:12:22.715521 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c4e992-ba4f-4d8a-8d42-6bef047bb0ac None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:22.716049 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c4e992-ba4f-4d8a-8d42-6bef047bb0ac None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:22.716526 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c4e992-ba4f-4d8a-8d42-6bef047bb0ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:22.732285 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-1928f2fe-cf9c-4d4d-bf8d-743c32c672e3 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:22.733884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1928f2fe-cf9c-4d4d-bf8d-743c32c672e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:22.747895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-e5c4e992-ba4f-4d8a-8d42-6bef047bb0ac None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:22.749505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e5c4e992-ba4f-4d8a-8d42-6bef047bb0ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:22.856686 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 58/112] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:22.858084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:22.866286 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 55/113] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:23.050261 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1724d5c7-7fd9-489f-aace-4d3e938fea22 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:23.051275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1724d5c7-7fd9-489f-aace-4d3e938fea22 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:23.051787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1724d5c7-7fd9-489f-aace-4d3e938fea22 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:23.053960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 59/114] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:23 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:23.059493 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4753de3-f9c7-4b63-8b6a-82d44cc11a3e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:23.060030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4753de3-f9c7-4b63-8b6a-82d44cc11a3e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:23.060511 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4753de3-f9c7-4b63-8b6a-82d44cc11a3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:23.088296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-a4753de3-f9c7-4b63-8b6a-82d44cc11a3e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:23.089858 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a4753de3-f9c7-4b63-8b6a-82d44cc11a3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:23.170928 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf579f0c-dcbb-4df9-bdc8-42c06ad046ae None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:23.171563 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf579f0c-dcbb-4df9-bdc8-42c06ad046ae None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:23.172066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf579f0c-dcbb-4df9-bdc8-42c06ad046ae None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:23.176951 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 60/115] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:23 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:12:23.182868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fe2f56-49bd-433e-8dae-367960cf74ec None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:23.183432 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fe2f56-49bd-433e-8dae-367960cf74ec None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:23.183911 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fe2f56-49bd-433e-8dae-367960cf74ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:23.215915 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-99fe2f56-49bd-433e-8dae-367960cf74ec None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:23.217941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-99fe2f56-49bd-433e-8dae-367960cf74ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:23.234490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 56/116] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:23.282790 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4753de3-f9c7-4b63-8b6a-82d44cc11a3e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:23.351444 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 61/117] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:23.387498 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fe2f56-49bd-433e-8dae-367960cf74ec None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:23.415167 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:23.445651 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4753de3-f9c7-4b63-8b6a-82d44cc11a3e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:23.500926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:23.520332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42851dc8-1460-4e1f-82ad-a97e630ed055 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:23.521367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42851dc8-1460-4e1f-82ad-a97e630ed055 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:23.521655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42851dc8-1460-4e1f-82ad-a97e630ed055 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:23.521931 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42851dc8-1460-4e1f-82ad-a97e630ed055 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:23.546368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42851dc8-1460-4e1f-82ad-a97e630ed055 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:23.552358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-42851dc8-1460-4e1f-82ad-a97e630ed055 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:23.552358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:23.552358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:23.552358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:23.552358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:23.552358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:23.552358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:23.552358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:23.552358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:23.552358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:23.552358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:23.552358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:23.552358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:23.552358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:23.552358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:23.552358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:23.553798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:23.553798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:23.553798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:23.553798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:23.553798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:23.553798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:23.553798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:23.553798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:23.553798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:23.553798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:23.553798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:23.553798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:23.553798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:23.553798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:23.553798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:23.553798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:23.554531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:23.554531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:23.554531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:23.554531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:23.554531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:23.554531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:23.554531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:23.554531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:23.554531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:23.554531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:23.554531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:23.554531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:23.554531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 57/118] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:12:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:12:23.563229 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42851dc8-1460-4e1f-82ad-a97e630ed055 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:23.580158 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fe2f56-49bd-433e-8dae-367960cf74ec None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:23.647360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ea5cca-b337-43b2-965e-8da5137e4f8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:23.647925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ea5cca-b337-43b2-965e-8da5137e4f8f None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:23.648207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ea5cca-b337-43b2-965e-8da5137e4f8f None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:23.648480 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ea5cca-b337-43b2-965e-8da5137e4f8f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:23.674004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ea5cca-b337-43b2-965e-8da5137e4f8f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:23.679095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-b2ea5cca-b337-43b2-965e-8da5137e4f8f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:23.679095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:23.679095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:23.679095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:23.679095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:23.679095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:23.679095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:23.679095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:23.679095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:23.679095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:23.679095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:23.679095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:23.679095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:23.679095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:23.679095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:23.679095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:23.679925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:23.679925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:23.679925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:23.679925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:23.679925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:23.679925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:23.679925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:23.679925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:23.679925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:23.679925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:23.679925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:23.679925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:23.679925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:23.679925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:23.679925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:23.679925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:23.680694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:23.680694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:23.680694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:23.680694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:23.680694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:23.680694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:23.680694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:23.680694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:23.680694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:23.680694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:23.680694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:23.680694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:23.680694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 62/119] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:23 2022] GET /identity/v3/roles/member => generated 84 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:12:23.687675 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ea5cca-b337-43b2-965e-8da5137e4f8f None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:23.796241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:23.801240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42851dc8-1460-4e1f-82ad-a97e630ed055 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:23.824538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ea5cca-b337-43b2-965e-8da5137e4f8f None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:23.837678 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7f5560-6393-4633-89c3-97645017191a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:23.838223 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7f5560-6393-4633-89c3-97645017191a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:23.838490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7f5560-6393-4633-89c3-97645017191a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:23.838765 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7f5560-6393-4633-89c3-97645017191a None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:23.839440 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7f5560-6393-4633-89c3-97645017191a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:23.839742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7f5560-6393-4633-89c3-97645017191a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:23.841278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7f5560-6393-4633-89c3-97645017191a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:23.850859 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 58/120] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:23.851731 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7f5560-6393-4633-89c3-97645017191a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:23.864039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa15c8a-246c-47db-a85a-d59ad489e1e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:23.865168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa15c8a-246c-47db-a85a-d59ad489e1e4 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:23.865670 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa15c8a-246c-47db-a85a-d59ad489e1e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:23.866135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa15c8a-246c-47db-a85a-d59ad489e1e4 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:23.867140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa15c8a-246c-47db-a85a-d59ad489e1e4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:23.867684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa15c8a-246c-47db-a85a-d59ad489e1e4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:23.869405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa15c8a-246c-47db-a85a-d59ad489e1e4 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:23.879243 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 63/121] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:23.880446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa15c8a-246c-47db-a85a-d59ad489e1e4 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:23.910408 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:23.967763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7f5560-6393-4633-89c3-97645017191a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:24.001721 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa15c8a-246c-47db-a85a-d59ad489e1e4 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:24.005379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486c9c54-8b9e-437f-bbf0-3a3fa8675df6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:24.006716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486c9c54-8b9e-437f-bbf0-3a3fa8675df6 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:24.007002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486c9c54-8b9e-437f-bbf0-3a3fa8675df6 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:24.007281 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486c9c54-8b9e-437f-bbf0-3a3fa8675df6 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:24.018310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486c9c54-8b9e-437f-bbf0-3a3fa8675df6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:24.024020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-486c9c54-8b9e-437f-bbf0-3a3fa8675df6 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:12:24.024020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:24.024020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:24.024020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:24.024020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:24.024020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:24.024020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:24.024020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:24.024020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:24.024020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:24.024020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:24.024020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:24.024020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:24.024020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:24.024020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:24.024020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:24.024844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:24.024844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:24.024844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:24.024844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:24.024844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:24.024844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:24.024844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:24.024844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:24.024844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:24.024844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:24.024844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:24.024844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:24.024844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:24.024844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:24.024844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:24.024844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:24.025652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:24.025652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:24.025652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:24.025652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:24.025652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:24.025652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:24.025652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:24.025652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:24.025652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:24.025652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:24.025652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:12:24.025652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:24.025652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 59/122] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:12:23 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:24.026351 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486c9c54-8b9e-437f-bbf0-3a3fa8675df6 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:24.045320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56eabda4-4f42-43eb-9953-f25119683cb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:24.046055 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56eabda4-4f42-43eb-9953-f25119683cb7 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:24.046340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56eabda4-4f42-43eb-9953-f25119683cb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:24.046614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56eabda4-4f42-43eb-9953-f25119683cb7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:24.054817 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:24.059604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56eabda4-4f42-43eb-9953-f25119683cb7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:24.065254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-56eabda4-4f42-43eb-9953-f25119683cb7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:24.065254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:24.065254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:24.065254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:24.065254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:24.065254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:24.065254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:24.065254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:24.065254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:24.065254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:24.065254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:24.065254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:24.065254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:24.065254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:24.065254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:24.065254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:24.066282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:24.066282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:24.066282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:24.066282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:24.066282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:24.066282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:24.066282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:24.066282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:24.066282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:24.066282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:24.066282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:24.066282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:24.066282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:24.066282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:24.066282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:24.066282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:24.067266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:24.067266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:24.067266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:24.067266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:24.067266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:24.067266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:24.067266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:24.067266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:24.067266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:24.067266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:24.067266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:24.067266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:24.069483 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 64/123] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:23 2022] GET /identity/v3/roles/member => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:24.074624 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56eabda4-4f42-43eb-9953-f25119683cb7 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:24.140028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486c9c54-8b9e-437f-bbf0-3a3fa8675df6 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:24.150815 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:24.178188 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2105601-d757-4a3b-b7cc-411ee13b442e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:24.178771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2105601-d757-4a3b-b7cc-411ee13b442e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:24.179051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2105601-d757-4a3b-b7cc-411ee13b442e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:24.179331 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2105601-d757-4a3b-b7cc-411ee13b442e None admin] PATH_INFO: `/v3/users/a85a3472aa4a4c95a2bc16b6450e8dcd` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:24.180474 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2105601-d757-4a3b-b7cc-411ee13b442e None admin] RBAC: Authorizing `identity:get_user(user_id=a85a3472aa4a4c95a2bc16b6450e8dcd)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:24.181976 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2105601-d757-4a3b-b7cc-411ee13b442e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:24.185163 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 60/124] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:24 2022] GET /identity/v3/users/a85a3472aa4a4c95a2bc16b6450e8dcd => generated 247 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:24.185163 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2105601-d757-4a3b-b7cc-411ee13b442e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:24.185337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56eabda4-4f42-43eb-9953-f25119683cb7 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:24.224397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaaa478d-faf9-49d5-8e6c-05d0de5c5635 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:24.225805 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaa478d-faf9-49d5-8e6c-05d0de5c5635 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:24.226159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaa478d-faf9-49d5-8e6c-05d0de5c5635 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:24.226499 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaa478d-faf9-49d5-8e6c-05d0de5c5635 None admin] PATH_INFO: `/v3/users/6a1eb60c46aa4d27b91657cd7f9fa60f` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:24.227712 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaaa478d-faf9-49d5-8e6c-05d0de5c5635 None admin] RBAC: Authorizing `identity:get_user(user_id=6a1eb60c46aa4d27b91657cd7f9fa60f)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:24.229255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaaa478d-faf9-49d5-8e6c-05d0de5c5635 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:24.231007 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 65/125] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:24 2022] GET /identity/v3/users/6a1eb60c46aa4d27b91657cd7f9fa60f => generated 275 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:24.231794 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaaa478d-faf9-49d5-8e6c-05d0de5c5635 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:24.296976 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2105601-d757-4a3b-b7cc-411ee13b442e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:24.300196 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:24.335404 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee56b2f-ee5d-44e5-a16c-36e78d29bbd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:24.336033 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee56b2f-ee5d-44e5-a16c-36e78d29bbd5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:24.336317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee56b2f-ee5d-44e5-a16c-36e78d29bbd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:24.336593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee56b2f-ee5d-44e5-a16c-36e78d29bbd5 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:24.337290 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee56b2f-ee5d-44e5-a16c-36e78d29bbd5 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:24.337600 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee56b2f-ee5d-44e5-a16c-36e78d29bbd5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:24.338687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaaa478d-faf9-49d5-8e6c-05d0de5c5635 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:24.338993 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee56b2f-ee5d-44e5-a16c-36e78d29bbd5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:24.347216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 61/126] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:12:24 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:24.350523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3ca83d-7e0b-4865-9160-e1f52dfc2f1b None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:24.351433 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3ca83d-7e0b-4865-9160-e1f52dfc2f1b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:24.352078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3ca83d-7e0b-4865-9160-e1f52dfc2f1b None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:24.357587 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 62/127] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:24 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:24.358382 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3ca83d-7e0b-4865-9160-e1f52dfc2f1b None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:24.376843 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a143bbb-6807-4ee1-baaf-0959aadf9055 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:24.377890 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a143bbb-6807-4ee1-baaf-0959aadf9055 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:24.378181 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a143bbb-6807-4ee1-baaf-0959aadf9055 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:24.378459 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a143bbb-6807-4ee1-baaf-0959aadf9055 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:24.379238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a143bbb-6807-4ee1-baaf-0959aadf9055 None admin] RBAC: Adding query filter params (name=member) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:24.379448 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a143bbb-6807-4ee1-baaf-0959aadf9055 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:24.382159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a143bbb-6807-4ee1-baaf-0959aadf9055 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:24.392778 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 66/128] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:24 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:24.397192 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a143bbb-6807-4ee1-baaf-0959aadf9055 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:24.406220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:24.471393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3ca83d-7e0b-4865-9160-e1f52dfc2f1b None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:24.502556 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a143bbb-6807-4ee1-baaf-0959aadf9055 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:24.508579 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae9428a-2476-49ae-a45d-8d8a59e3cc2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:24.509250 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae9428a-2476-49ae-a45d-8d8a59e3cc2b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:24.509545 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae9428a-2476-49ae-a45d-8d8a59e3cc2b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:24.509823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae9428a-2476-49ae-a45d-8d8a59e3cc2b None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:24.511050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae9428a-2476-49ae-a45d-8d8a59e3cc2b None admin] RBAC: Authorizing `identity:get_project(project_id=26661cf5060b4b939d8ab02f9ea45daf)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:24.512544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae9428a-2476-49ae-a45d-8d8a59e3cc2b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:24.518782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 63/129] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:24 2022] GET /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf => generated 298 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:24.518782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae9428a-2476-49ae-a45d-8d8a59e3cc2b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:24.535144 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:24.545503 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77262dd-808c-4d81-ab82-69a1b3894615 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:24.546305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77262dd-808c-4d81-ab82-69a1b3894615 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:24.546591 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77262dd-808c-4d81-ab82-69a1b3894615 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:24.547196 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77262dd-808c-4d81-ab82-69a1b3894615 None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:24.548598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77262dd-808c-4d81-ab82-69a1b3894615 None admin] RBAC: Authorizing `identity:get_project(project_id=26661cf5060b4b939d8ab02f9ea45daf)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:24.550834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77262dd-808c-4d81-ab82-69a1b3894615 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:24.557303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 67/130] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:24 2022] GET /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf => generated 298 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:24.557303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdaaf3d-1c9e-489f-8886-8baef558e7ae None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:24.557303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdaaf3d-1c9e-489f-8886-8baef558e7ae None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:24.557303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdaaf3d-1c9e-489f-8886-8baef558e7ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:24.578193 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-4fdaaf3d-1c9e-489f-8886-8baef558e7ae None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:24.579699 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4fdaaf3d-1c9e-489f-8886-8baef558e7ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:24.621343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae9428a-2476-49ae-a45d-8d8a59e3cc2b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:24.628289 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:24.657668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f978fab0-2d6d-46c0-8e62-e8993c363151 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:24.658302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f978fab0-2d6d-46c0-8e62-e8993c363151 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:24.658579 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f978fab0-2d6d-46c0-8e62-e8993c363151 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:24.658843 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 68/131] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:24.659189 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f978fab0-2d6d-46c0-8e62-e8993c363151 None admin] PATH_INFO: `/v3/users/6a1eb60c46aa4d27b91657cd7f9fa60f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:24.660477 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f978fab0-2d6d-46c0-8e62-e8993c363151 None admin] RBAC: Authorizing `identity:get_user(user_id=6a1eb60c46aa4d27b91657cd7f9fa60f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:24.662007 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f978fab0-2d6d-46c0-8e62-e8993c363151 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:24.662229 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdaaf3d-1c9e-489f-8886-8baef558e7ae None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:24.663712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 64/132] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:24 2022] GET /identity/v3/users/6a1eb60c46aa4d27b91657cd7f9fa60f => generated 275 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:24.665833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f978fab0-2d6d-46c0-8e62-e8993c363151 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:24.778948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f978fab0-2d6d-46c0-8e62-e8993c363151 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:24.780080 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdaaf3d-1c9e-489f-8886-8baef558e7ae None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:24.807257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:24.816867 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86a67a5-bc41-4b98-86e4-41925df9f853 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:24.817019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7062e8-9cba-46f5-b2e3-d54c830243ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:24.817448 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86a67a5-bc41-4b98-86e4-41925df9f853 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:24.817548 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7062e8-9cba-46f5-b2e3-d54c830243ae None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:24.817748 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86a67a5-bc41-4b98-86e4-41925df9f853 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:24.817843 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7062e8-9cba-46f5-b2e3-d54c830243ae None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:24.818006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86a67a5-bc41-4b98-86e4-41925df9f853 None admin] PATH_INFO: `/v3/users/6a1eb60c46aa4d27b91657cd7f9fa60f` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:24.818107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7062e8-9cba-46f5-b2e3-d54c830243ae None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf/users/a85a3472aa4a4c95a2bc16b6450e8dcd/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:24.819701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86a67a5-bc41-4b98-86e4-41925df9f853 None admin] RBAC: Authorizing `identity:get_user(user_id=6a1eb60c46aa4d27b91657cd7f9fa60f)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:24.820523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7062e8-9cba-46f5-b2e3-d54c830243ae None admin] RBAC: Authorizing `identity:create_grant(project_id=26661cf5060b4b939d8ab02f9ea45daf, user_id=a85a3472aa4a4c95a2bc16b6450e8dcd, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:24.821140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86a67a5-bc41-4b98-86e4-41925df9f853 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:24.821976 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7062e8-9cba-46f5-b2e3-d54c830243ae None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:24.822713 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 69/133] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:24 2022] GET /identity/v3/users/6a1eb60c46aa4d27b91657cd7f9fa60f => generated 275 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:24.827977 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86a67a5-bc41-4b98-86e4-41925df9f853 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:24.835272 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 65/134] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:12:24 2022] PUT /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf/users/a85a3472aa4a4c95a2bc16b6450e8dcd/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:12:24.836280 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7062e8-9cba-46f5-b2e3-d54c830243ae None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:24.882898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:24.943274 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86a67a5-bc41-4b98-86e4-41925df9f853 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:24.965247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7062e8-9cba-46f5-b2e3-d54c830243ae None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:24.984536 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02ed66b-f935-419a-b61d-84cf6629c3fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:24.985253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02ed66b-f935-419a-b61d-84cf6629c3fd None admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:24.985534 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02ed66b-f935-419a-b61d-84cf6629c3fd None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:24.985807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02ed66b-f935-419a-b61d-84cf6629c3fd None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf/users/6a1eb60c46aa4d27b91657cd7f9fa60f/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:24.987738 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02ed66b-f935-419a-b61d-84cf6629c3fd None admin] RBAC: Authorizing `identity:create_grant(project_id=26661cf5060b4b939d8ab02f9ea45daf, user_id=6a1eb60c46aa4d27b91657cd7f9fa60f, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:24.989283 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02ed66b-f935-419a-b61d-84cf6629c3fd None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:25.005802 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d333c4a1-72c3-47a7-a248-5d81490632f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:25.006727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d333c4a1-72c3-47a7-a248-5d81490632f0 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:25.007017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d333c4a1-72c3-47a7-a248-5d81490632f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:25.007311 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 70/135] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:12:24 2022] PUT /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf/users/6a1eb60c46aa4d27b91657cd7f9fa60f/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:12:25.008558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe075a18-225e-43f6-aab9-9ab01f916cb5 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:25.008847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe075a18-225e-43f6-aab9-9ab01f916cb5 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:25.009159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe075a18-225e-43f6-aab9-9ab01f916cb5 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:25.010603 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 71/136] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:25.010710 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d333c4a1-72c3-47a7-a248-5d81490632f0 None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:25.011142 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe075a18-225e-43f6-aab9-9ab01f916cb5 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:25.012300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d333c4a1-72c3-47a7-a248-5d81490632f0 None admin] RBAC: Authorizing `identity:get_project(project_id=26661cf5060b4b939d8ab02f9ea45daf)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:25.013954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d333c4a1-72c3-47a7-a248-5d81490632f0 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:25.021278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 66/137] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:24 2022] GET /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf => generated 298 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:25.021278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d333c4a1-72c3-47a7-a248-5d81490632f0 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:25.101631 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:25.146964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d333c4a1-72c3-47a7-a248-5d81490632f0 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:25.148794 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:25.169462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe075a18-225e-43f6-aab9-9ab01f916cb5 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:25.191185 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495479c2-b342-4dda-8450-66fed962917d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:25.192119 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495479c2-b342-4dda-8450-66fed962917d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:25.192244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495479c2-b342-4dda-8450-66fed962917d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:25.192443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495479c2-b342-4dda-8450-66fed962917d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:25.194112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495479c2-b342-4dda-8450-66fed962917d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:25.196000 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495479c2-b342-4dda-8450-66fed962917d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:25.198543 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 67/138] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:12:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:25.199495 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407c3d71-9e7f-442a-ad11-c8568260c9b2 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:25.199495 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407c3d71-9e7f-442a-ad11-c8568260c9b2 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:25.199697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407c3d71-9e7f-442a-ad11-c8568260c9b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:25.213440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560d264d-b8b9-456e-ae6a-98556a6827fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:25.214077 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560d264d-b8b9-456e-ae6a-98556a6827fa None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:25.215184 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560d264d-b8b9-456e-ae6a-98556a6827fa None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:25.215184 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560d264d-b8b9-456e-ae6a-98556a6827fa None admin] PATH_INFO: `/v3/projects/c20f0dd259eb4d55856b1f0848964837` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:25.215853 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560d264d-b8b9-456e-ae6a-98556a6827fa None admin] RBAC: Authorizing `identity:get_project(project_id=c20f0dd259eb4d55856b1f0848964837)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:25.217732 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560d264d-b8b9-456e-ae6a-98556a6827fa None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:25.218967 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 72/139] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:25 2022] GET /identity/v3/projects/c20f0dd259eb4d55856b1f0848964837 => generated 312 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:25.223370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-407c3d71-9e7f-442a-ad11-c8568260c9b2 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:25.224199 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560d264d-b8b9-456e-ae6a-98556a6827fa None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:25.225202 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-407c3d71-9e7f-442a-ad11-c8568260c9b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:25.301959 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 68/140] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:25.303007 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407c3d71-9e7f-442a-ad11-c8568260c9b2 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:25.358137 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560d264d-b8b9-456e-ae6a-98556a6827fa None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:25.385469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:25.397182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f6fe53-d271-4ff1-8de4-9ad02d9cf348 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:25.397956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f6fe53-d271-4ff1-8de4-9ad02d9cf348 None admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:25.398245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f6fe53-d271-4ff1-8de4-9ad02d9cf348 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:25.398521 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f6fe53-d271-4ff1-8de4-9ad02d9cf348 None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf/users/6a1eb60c46aa4d27b91657cd7f9fa60f/roles/a9e862ac528245ca8363bccc5bf35287` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:25.400900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f6fe53-d271-4ff1-8de4-9ad02d9cf348 None admin] RBAC: Authorizing `identity:create_grant(project_id=26661cf5060b4b939d8ab02f9ea45daf, user_id=6a1eb60c46aa4d27b91657cd7f9fa60f, role_id=a9e862ac528245ca8363bccc5bf35287)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:25.402536 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f6fe53-d271-4ff1-8de4-9ad02d9cf348 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:25.407965 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:25.411368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407c3d71-9e7f-442a-ad11-c8568260c9b2 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:25.417852 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 73/141] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:12:25 2022] PUT /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf/users/6a1eb60c46aa4d27b91657cd7f9fa60f/roles/a9e862ac528245ca8363bccc5bf35287 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:12:25.418925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f6fe53-d271-4ff1-8de4-9ad02d9cf348 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:25.471912 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737c425b-afce-4ae7-b6d9-2e0b818a345b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:25.472617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737c425b-afce-4ae7-b6d9-2e0b818a345b None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:25.472834 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737c425b-afce-4ae7-b6d9-2e0b818a345b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:25.473169 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737c425b-afce-4ae7-b6d9-2e0b818a345b None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:25.473925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737c425b-afce-4ae7-b6d9-2e0b818a345b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:25.476032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737c425b-afce-4ae7-b6d9-2e0b818a345b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:25.519813 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 69/142] 10.222.0.85 () {60 vars in 1132 bytes} [Tue Aug 30 04:12:25 2022] POST /identity/v3/users => generated 283 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:12:25.547767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f6fe53-d271-4ff1-8de4-9ad02d9cf348 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:25.597883 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9c72a3-cdb3-4e90-a068-4a09b77a71cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:25.598762 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9c72a3-cdb3-4e90-a068-4a09b77a71cd None admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:25.599145 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9c72a3-cdb3-4e90-a068-4a09b77a71cd None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:25.600192 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9c72a3-cdb3-4e90-a068-4a09b77a71cd None admin] PATH_INFO: `/v3/projects/c20f0dd259eb4d55856b1f0848964837/users/6a1eb60c46aa4d27b91657cd7f9fa60f/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:25.602379 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9c72a3-cdb3-4e90-a068-4a09b77a71cd None admin] RBAC: Authorizing `identity:create_grant(project_id=c20f0dd259eb4d55856b1f0848964837, user_id=6a1eb60c46aa4d27b91657cd7f9fa60f, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:25.604039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9c72a3-cdb3-4e90-a068-4a09b77a71cd None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:25.623994 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 74/143] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:12:25 2022] PUT /identity/v3/projects/c20f0dd259eb4d55856b1f0848964837/users/6a1eb60c46aa4d27b91657cd7f9fa60f/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:12:25.730515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737c425b-afce-4ae7-b6d9-2e0b818a345b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:25.859406 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737c425b-afce-4ae7-b6d9-2e0b818a345b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:25.899655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455ccad9-5a0b-4da1-800f-e186581b4692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:25.900398 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455ccad9-5a0b-4da1-800f-e186581b4692 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:25.900773 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455ccad9-5a0b-4da1-800f-e186581b4692 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:25.901277 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455ccad9-5a0b-4da1-800f-e186581b4692 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:25.920848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455ccad9-5a0b-4da1-800f-e186581b4692 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:25.926833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-455ccad9-5a0b-4da1-800f-e186581b4692 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:12:25.926833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:25.926833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:25.926833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:25.926833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:25.926833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:25.926833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:25.926833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:25.926833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:25.926833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:25.926833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:25.926833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:25.926833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:25.926833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:25.926833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:25.926833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:25.928400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:25.928400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:25.928400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:25.928400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:25.928400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:25.928400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:25.928400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:25.928400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:25.928400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:25.928400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:25.928400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:25.928400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:25.928400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:25.928400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:25.928400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:25.928400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:25.929360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:25.929360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:25.929360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:25.929360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:25.929360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:25.929360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:25.929360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:25.929360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:25.929360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:25.929360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:25.929360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:12:25.929360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:25.931350 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 70/144] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:25 2022] GET /identity/v3/roles/reader => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:25.937249 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9c72a3-cdb3-4e90-a068-4a09b77a71cd None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:25.987745 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:26.064285 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9c72a3-cdb3-4e90-a068-4a09b77a71cd None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:26.104573 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774cd03e-e6b9-4b1d-a39d-3be8950b4ba1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:26.105245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774cd03e-e6b9-4b1d-a39d-3be8950b4ba1 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:26.105491 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774cd03e-e6b9-4b1d-a39d-3be8950b4ba1 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:26.105766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774cd03e-e6b9-4b1d-a39d-3be8950b4ba1 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:26.108790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774cd03e-e6b9-4b1d-a39d-3be8950b4ba1 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:26.108790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774cd03e-e6b9-4b1d-a39d-3be8950b4ba1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:26.108790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774cd03e-e6b9-4b1d-a39d-3be8950b4ba1 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:26.124245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 75/145] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:25 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:26.130096 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455ccad9-5a0b-4da1-800f-e186581b4692 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:26.151277 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:26.241969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455ccad9-5a0b-4da1-800f-e186581b4692 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:26.280117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d34794-a78b-4521-8558-40d5b5cd9d54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:26.280762 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d34794-a78b-4521-8558-40d5b5cd9d54 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:26.281065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d34794-a78b-4521-8558-40d5b5cd9d54 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:26.281340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d34794-a78b-4521-8558-40d5b5cd9d54 None admin] PATH_INFO: `/v3/users/3124ffc09dfd4a52880634dbc9fcad43` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:26.282479 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d34794-a78b-4521-8558-40d5b5cd9d54 None admin] RBAC: Authorizing `identity:get_user(user_id=3124ffc09dfd4a52880634dbc9fcad43)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:26.283955 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d34794-a78b-4521-8558-40d5b5cd9d54 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:26.285972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 71/146] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:26 2022] GET /identity/v3/users/3124ffc09dfd4a52880634dbc9fcad43 => generated 289 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:26.291860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774cd03e-e6b9-4b1d-a39d-3be8950b4ba1 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:26.435064 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774cd03e-e6b9-4b1d-a39d-3be8950b4ba1 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:26.442230 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:26.484468 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c673a22d-ab6a-4cde-934d-dad203210ac1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:26.485134 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c673a22d-ab6a-4cde-934d-dad203210ac1 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:26.485414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c673a22d-ab6a-4cde-934d-dad203210ac1 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:26.485681 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c673a22d-ab6a-4cde-934d-dad203210ac1 None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:26.486817 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c673a22d-ab6a-4cde-934d-dad203210ac1 None admin] RBAC: Authorizing `identity:get_project(project_id=26661cf5060b4b939d8ab02f9ea45daf)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:26.488343 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c673a22d-ab6a-4cde-934d-dad203210ac1 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:26.490009 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 76/147] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:26 2022] GET /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf => generated 298 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:26.496307 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d34794-a78b-4521-8558-40d5b5cd9d54 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:26.607297 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:26.626770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d34794-a78b-4521-8558-40d5b5cd9d54 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:26.664641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a215480-cae4-4cec-9599-82dee6a6d12a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:26.665363 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a215480-cae4-4cec-9599-82dee6a6d12a None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:26.665646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a215480-cae4-4cec-9599-82dee6a6d12a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:26.665926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a215480-cae4-4cec-9599-82dee6a6d12a None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf/users/3124ffc09dfd4a52880634dbc9fcad43/roles/902fd8d9a4ef48058e1b17e0429ede58` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:26.667860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a215480-cae4-4cec-9599-82dee6a6d12a None admin] RBAC: Authorizing `identity:create_grant(project_id=26661cf5060b4b939d8ab02f9ea45daf, user_id=3124ffc09dfd4a52880634dbc9fcad43, role_id=902fd8d9a4ef48058e1b17e0429ede58)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:26.669350 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a215480-cae4-4cec-9599-82dee6a6d12a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:26.689566 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 72/148] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:12:26 2022] PUT /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf/users/3124ffc09dfd4a52880634dbc9fcad43/roles/902fd8d9a4ef48058e1b17e0429ede58 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:12:27.728821 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b11852-b511-4b66-a7a9-32bbc2544048 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:27.729381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b11852-b511-4b66-a7a9-32bbc2544048 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:27.729381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b11852-b511-4b66-a7a9-32bbc2544048 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:27.730877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 77/149] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:27.745773 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4197490-0fef-4842-9ee7-d5e6cfc0e65b None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:27.746260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4197490-0fef-4842-9ee7-d5e6cfc0e65b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:27.746330 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4197490-0fef-4842-9ee7-d5e6cfc0e65b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:27.774159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b4197490-0fef-4842-9ee7-d5e6cfc0e65b None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:27.775372 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b4197490-0fef-4842-9ee7-d5e6cfc0e65b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:27.855586 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 73/150] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:27.926817 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c9e6d2-0fd8-4ee4-9a08-d5543d653c8d None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:27.927117 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c9e6d2-0fd8-4ee4-9a08-d5543d653c8d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:27.927360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c9e6d2-0fd8-4ee4-9a08-d5543d653c8d None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:27.929424 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 78/151] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:27.935775 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91023265-a49c-4ed5-a87b-8fa84a698702 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:27.935775 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91023265-a49c-4ed5-a87b-8fa84a698702 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:27.935775 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91023265-a49c-4ed5-a87b-8fa84a698702 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:27.962062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-91023265-a49c-4ed5-a87b-8fa84a698702 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:27.963257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-91023265-a49c-4ed5-a87b-8fa84a698702 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:28.039866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 74/152] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:28.332219 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c9e6d2-0fd8-4ee4-9a08-d5543d653c8d None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:28.451333 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c9e6d2-0fd8-4ee4-9a08-d5543d653c8d None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:28.484845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8a64ec-1316-4a9c-8cbd-10d26c90b9f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:28.485507 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8a64ec-1316-4a9c-8cbd-10d26c90b9f7 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:28.485781 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8a64ec-1316-4a9c-8cbd-10d26c90b9f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:28.486048 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8a64ec-1316-4a9c-8cbd-10d26c90b9f7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:28.497722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8a64ec-1316-4a9c-8cbd-10d26c90b9f7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:28.502513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-eb8a64ec-1316-4a9c-8cbd-10d26c90b9f7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:28.502513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:28.502513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:28.502513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:28.502513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:28.502513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:28.502513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:28.502513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:28.502513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:28.502513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:28.502513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:28.502513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:28.502513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:28.502513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:28.502513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:28.502513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:28.503422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:28.503422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:28.503422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:28.503422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:28.503422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:28.503422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:28.503422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:28.503422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:28.503422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:28.503422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:28.503422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:28.503422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:28.503422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:28.503422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:28.503422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:28.503422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:28.504183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:28.504183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:28.504183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:28.504183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:28.504183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:28.504183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:28.504183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:28.504183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:28.504183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:28.504183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:28.504183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:28.504183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:28.504183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 79/153] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:28.511629 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91023265-a49c-4ed5-a87b-8fa84a698702 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:28.515214 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:28.630373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91023265-a49c-4ed5-a87b-8fa84a698702 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:28.672757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8a64ec-1316-4a9c-8cbd-10d26c90b9f7 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:28.694172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab8bdb9-fe18-483c-b35a-bf1a77d803d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:28.694804 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab8bdb9-fe18-483c-b35a-bf1a77d803d8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:28.695079 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab8bdb9-fe18-483c-b35a-bf1a77d803d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:28.695345 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab8bdb9-fe18-483c-b35a-bf1a77d803d8 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:28.696026 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab8bdb9-fe18-483c-b35a-bf1a77d803d8 None admin] RBAC: Adding query filter params (name=member) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:28.696321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab8bdb9-fe18-483c-b35a-bf1a77d803d8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:28.697739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab8bdb9-fe18-483c-b35a-bf1a77d803d8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:28.717428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:28.719437 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 75/154] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:28.721818 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc1a788-81df-4509-8f0d-ba82a3eedc48 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:28.722289 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc1a788-81df-4509-8f0d-ba82a3eedc48 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:28.722720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc1a788-81df-4509-8f0d-ba82a3eedc48 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:28.724333 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 76/155] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:28.730363 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6418d9ca-c984-4ba3-904e-446051c2b1c2 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:28.730822 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6418d9ca-c984-4ba3-904e-446051c2b1c2 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:28.731264 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6418d9ca-c984-4ba3-904e-446051c2b1c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:28.770216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-6418d9ca-c984-4ba3-904e-446051c2b1c2 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:28.771464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6418d9ca-c984-4ba3-904e-446051c2b1c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:28.835276 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8a64ec-1316-4a9c-8cbd-10d26c90b9f7 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:28.877145 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97220e65-6b15-4778-8b04-1a29b88cf21d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:28.878267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97220e65-6b15-4778-8b04-1a29b88cf21d None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:28.878690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97220e65-6b15-4778-8b04-1a29b88cf21d None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:28.879098 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97220e65-6b15-4778-8b04-1a29b88cf21d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:28.892437 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97220e65-6b15-4778-8b04-1a29b88cf21d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:28.898737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-97220e65-6b15-4778-8b04-1a29b88cf21d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:28.898737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:28.898737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:28.898737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:28.898737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:28.898737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:28.898737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:28.898737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:28.898737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:28.898737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:28.898737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:28.898737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:28.898737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:28.898737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:28.898737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:28.898737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:28.899913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:28.899913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:28.899913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:28.899913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:28.899913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:28.899913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:28.899913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:28.899913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:28.899913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:28.899913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:28.899913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:28.899913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:28.899913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:28.899913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:28.899913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:28.899913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:28.900809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:28.900809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:28.900809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:28.900809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:28.900809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:28.900809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:28.900809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:28.900809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:28.900809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:28.900809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:28.900809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:28.900809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:28.903702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 80/156] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:12:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:12:28.904713 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97220e65-6b15-4778-8b04-1a29b88cf21d None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:28.909763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:28.927478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 77/157] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:28.929168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6418d9ca-c984-4ba3-904e-446051c2b1c2 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:29.016312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:29.027245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97220e65-6b15-4778-8b04-1a29b88cf21d None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:29.067534 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76790f2f-bde5-420f-9d1b-17c8fff7bde7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:29.069346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76790f2f-bde5-420f-9d1b-17c8fff7bde7 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:29.069817 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76790f2f-bde5-420f-9d1b-17c8fff7bde7 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:29.070450 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76790f2f-bde5-420f-9d1b-17c8fff7bde7 None admin] PATH_INFO: `/v3/users/6a1eb60c46aa4d27b91657cd7f9fa60f` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:29.071762 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76790f2f-bde5-420f-9d1b-17c8fff7bde7 None admin] RBAC: Authorizing `identity:get_user(user_id=6a1eb60c46aa4d27b91657cd7f9fa60f)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:29.073413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76790f2f-bde5-420f-9d1b-17c8fff7bde7 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:29.076157 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 81/158] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:28 2022] GET /identity/v3/users/6a1eb60c46aa4d27b91657cd7f9fa60f => generated 275 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:29.083117 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76790f2f-bde5-420f-9d1b-17c8fff7bde7 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:29.138391 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6418d9ca-c984-4ba3-904e-446051c2b1c2 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:29.204450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1675ca-bea0-40a4-978f-9f559226f613 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:29.205084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1675ca-bea0-40a4-978f-9f559226f613 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:29.206224 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1675ca-bea0-40a4-978f-9f559226f613 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:29.206597 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1675ca-bea0-40a4-978f-9f559226f613 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:29.207288 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1675ca-bea0-40a4-978f-9f559226f613 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:29.210128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1675ca-bea0-40a4-978f-9f559226f613 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:29.211547 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1675ca-bea0-40a4-978f-9f559226f613 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:29.230541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 78/159] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:29.231842 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da46575-fd5d-4d5a-ab1d-737ced3a1d14 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:29.232945 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da46575-fd5d-4d5a-ab1d-737ced3a1d14 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:29.233262 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da46575-fd5d-4d5a-ab1d-737ced3a1d14 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:29.234789 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 79/160] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:29.236864 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da46575-fd5d-4d5a-ab1d-737ced3a1d14 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:29.275149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76790f2f-bde5-420f-9d1b-17c8fff7bde7 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:29.342366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:29.354915 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96f1886-440f-48bb-95f1-2be5a080aa52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:29.355533 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96f1886-440f-48bb-95f1-2be5a080aa52 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:29.355814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96f1886-440f-48bb-95f1-2be5a080aa52 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:29.356082 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96f1886-440f-48bb-95f1-2be5a080aa52 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:29.404351 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da46575-fd5d-4d5a-ab1d-737ced3a1d14 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:29.429616 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:29.474816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280068b5-ec6c-4e0c-b975-4c37a27029fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:29.476320 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280068b5-ec6c-4e0c-b975-4c37a27029fd None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:29.477241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280068b5-ec6c-4e0c-b975-4c37a27029fd None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:29.477884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280068b5-ec6c-4e0c-b975-4c37a27029fd None admin] PATH_INFO: `/v3/users/a85a3472aa4a4c95a2bc16b6450e8dcd` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:29.481869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280068b5-ec6c-4e0c-b975-4c37a27029fd None admin] RBAC: Authorizing `identity:get_user(user_id=a85a3472aa4a4c95a2bc16b6450e8dcd)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:29.483921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280068b5-ec6c-4e0c-b975-4c37a27029fd None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:29.485821 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 80/161] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:29 2022] GET /identity/v3/users/a85a3472aa4a4c95a2bc16b6450e8dcd => generated 247 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:29.487827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d75099-2140-4039-a556-25f49a2d4646 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:29.488421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d75099-2140-4039-a556-25f49a2d4646 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:29.488909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d75099-2140-4039-a556-25f49a2d4646 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:29.528373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-13d75099-2140-4039-a556-25f49a2d4646 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:29.530021 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96f1886-440f-48bb-95f1-2be5a080aa52 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:29.530357 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-13d75099-2140-4039-a556-25f49a2d4646 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:29.531547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96f1886-440f-48bb-95f1-2be5a080aa52 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:29.665758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 81/162] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:29.666950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d75099-2140-4039-a556-25f49a2d4646 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:29.671647 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 82/163] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 589 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:12:29.672523 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96f1886-440f-48bb-95f1-2be5a080aa52 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:29.742885 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:29.772966 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:29.792162 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96f1886-440f-48bb-95f1-2be5a080aa52 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:29.835057 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcab976-68d1-49c4-8b0d-afb9fc6c3548 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:29.835701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcab976-68d1-49c4-8b0d-afb9fc6c3548 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:29.836299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcab976-68d1-49c4-8b0d-afb9fc6c3548 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:29.836299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcab976-68d1-49c4-8b0d-afb9fc6c3548 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:29.848638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcab976-68d1-49c4-8b0d-afb9fc6c3548 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:29.855078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-5bcab976-68d1-49c4-8b0d-afb9fc6c3548 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:29.855078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:29.855078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:29.855078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:29.855078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:29.855078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:29.855078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:29.855078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:29.855078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:29.855078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:29.855078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:29.855078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:29.855078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:29.855078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:29.855078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:29.855078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:29.855979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:29.855979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:29.855979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:29.855979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:29.855979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:29.855979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:29.855979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:29.855979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:29.855979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:29.855979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:29.855979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:29.855979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:29.855979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:29.855979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:29.855979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:29.855979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:29.856741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:29.856741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:29.856741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:29.856741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:29.856741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:29.856741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:29.856741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:29.856741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:29.856741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:29.856741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:29.856741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:29.856741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:29.856741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 83/164] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:29 2022] GET /identity/v3/roles/member => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:29.859944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcab976-68d1-49c4-8b0d-afb9fc6c3548 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:29.867323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d75099-2140-4039-a556-25f49a2d4646 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:29.950868 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57679202-dd37-4377-8b2d-e72154ac676f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:29.952193 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57679202-dd37-4377-8b2d-e72154ac676f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:29.953198 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57679202-dd37-4377-8b2d-e72154ac676f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:29.953950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57679202-dd37-4377-8b2d-e72154ac676f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:29.990585 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcab976-68d1-49c4-8b0d-afb9fc6c3548 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:30.014804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:30.031081 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1282af1-cb89-44dd-8034-70b957afe00a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:30.031721 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1282af1-cb89-44dd-8034-70b957afe00a None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:30.032252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1282af1-cb89-44dd-8034-70b957afe00a None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:30.032532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1282af1-cb89-44dd-8034-70b957afe00a None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:30.033696 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1282af1-cb89-44dd-8034-70b957afe00a None admin] RBAC: Authorizing `identity:get_project(project_id=26661cf5060b4b939d8ab02f9ea45daf)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:30.036172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1282af1-cb89-44dd-8034-70b957afe00a None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:30.038834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 84/165] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:29 2022] GET /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf => generated 298 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:30.040165 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1282af1-cb89-44dd-8034-70b957afe00a None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:30.074197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57679202-dd37-4377-8b2d-e72154ac676f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:30.075731 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57679202-dd37-4377-8b2d-e72154ac676f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:30.141932 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:30.169948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1282af1-cb89-44dd-8034-70b957afe00a None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:30.207087 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 82/166] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 540 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:12:30.207271 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73b6007-d358-49fb-94eb-33717f3971ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:30.207811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73b6007-d358-49fb-94eb-33717f3971ed None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:30.208091 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73b6007-d358-49fb-94eb-33717f3971ed None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:30.208455 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73b6007-d358-49fb-94eb-33717f3971ed None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:30.208558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacf472d-fc62-4789-bcd0-56aa1d8d147d None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:30.208748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacf472d-fc62-4789-bcd0-56aa1d8d147d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:30.209571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacf472d-fc62-4789-bcd0-56aa1d8d147d None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:30.209750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73b6007-d358-49fb-94eb-33717f3971ed None admin] RBAC: Adding query filter params (name=member) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:30.210034 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73b6007-d358-49fb-94eb-33717f3971ed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:30.210826 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 83/167] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:30.211492 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73b6007-d358-49fb-94eb-33717f3971ed None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:30.211608 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacf472d-fc62-4789-bcd0-56aa1d8d147d None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:30.231259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 85/168] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:30 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:30.232132 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73b6007-d358-49fb-94eb-33717f3971ed None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:30.341438 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:30.379370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacf472d-fc62-4789-bcd0-56aa1d8d147d None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:30.437028 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73b6007-d358-49fb-94eb-33717f3971ed None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:30.443446 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46505c4d-3350-4cee-ad9a-5218d663423b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:30.444066 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46505c4d-3350-4cee-ad9a-5218d663423b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:30.444341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46505c4d-3350-4cee-ad9a-5218d663423b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:30.444608 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46505c4d-3350-4cee-ad9a-5218d663423b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:30.445375 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46505c4d-3350-4cee-ad9a-5218d663423b None admin] RBAC: Adding query filter params (role.id=ae0cef8120cf4dfc899c30803243a851, scope.project.id=26661cf5060b4b939d8ab02f9ea45daf, user.id=6a1eb60c46aa4d27b91657cd7f9fa60f) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:30.445678 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46505c4d-3350-4cee-ad9a-5218d663423b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:30.447137 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46505c4d-3350-4cee-ad9a-5218d663423b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:30.463194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 84/169] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:12:30 2022] GET /identity/v3/role_assignments?user.id=6a1eb60c46aa4d27b91657cd7f9fa60f&scope.project.id=26661cf5060b4b939d8ab02f9ea45daf&role.id=ae0cef8120cf4dfc899c30803243a851 => generated 608 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:30.464093 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46505c4d-3350-4cee-ad9a-5218d663423b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:30.521928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:30.523537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75189055-6612-42c7-be7d-a06f26df430e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:30.524350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75189055-6612-42c7-be7d-a06f26df430e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:30.524683 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75189055-6612-42c7-be7d-a06f26df430e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:30.524983 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75189055-6612-42c7-be7d-a06f26df430e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:30.545404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75189055-6612-42c7-be7d-a06f26df430e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:30.551035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-75189055-6612-42c7-be7d-a06f26df430e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:12:30.551035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:30.551035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:30.551035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:30.551035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:30.551035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:30.551035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:30.551035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:30.551035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:30.551035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:30.551035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:30.551035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:30.551035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:30.551035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:30.551035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:30.551035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:30.551985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:30.551985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:30.551985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:30.551985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:30.551985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:30.551985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:30.551985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:30.551985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:30.551985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:30.551985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:30.551985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:30.551985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:30.551985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:30.551985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:30.551985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:30.551985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:30.553401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:30.553401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:30.553401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:30.553401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:30.553401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:30.553401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:30.553401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:30.553401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:30.553401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:30.553401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:30.553401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:12:30.553401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:30.553401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 86/170] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:12:30 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 321 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:30.559279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35705e01-d988-48b9-a01d-7da52b8cdeb2 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:30.559279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35705e01-d988-48b9-a01d-7da52b8cdeb2 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:30.559279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35705e01-d988-48b9-a01d-7da52b8cdeb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:30.574386 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-35705e01-d988-48b9-a01d-7da52b8cdeb2 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:30.575639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-35705e01-d988-48b9-a01d-7da52b8cdeb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:30.655930 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 87/171] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:30.656742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35705e01-d988-48b9-a01d-7da52b8cdeb2 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:30.682441 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46505c4d-3350-4cee-ad9a-5218d663423b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:30.711692 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:30.751098 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27685ce3-0bb7-48cc-a7f3-ac79a83daffc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:30.751742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27685ce3-0bb7-48cc-a7f3-ac79a83daffc None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:30.752017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27685ce3-0bb7-48cc-a7f3-ac79a83daffc None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:30.753060 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27685ce3-0bb7-48cc-a7f3-ac79a83daffc None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:30.754984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27685ce3-0bb7-48cc-a7f3-ac79a83daffc None admin] RBAC: Authorizing `identity:get_project(project_id=26661cf5060b4b939d8ab02f9ea45daf)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:30.759167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27685ce3-0bb7-48cc-a7f3-ac79a83daffc None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:30.763195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 85/172] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:30 2022] GET /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf => generated 298 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:30.764391 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e03f93-25cb-4820-88da-1222b9a550b4 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:30.764673 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e03f93-25cb-4820-88da-1222b9a550b4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:30.764949 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e03f93-25cb-4820-88da-1222b9a550b4 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:30.767756 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 86/173] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:30.768457 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e03f93-25cb-4820-88da-1222b9a550b4 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:30.823260 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35705e01-d988-48b9-a01d-7da52b8cdeb2 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:30.873063 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae7f8f5-d81b-4cff-b562-37cc61fb3af2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:30.874553 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae7f8f5-d81b-4cff-b562-37cc61fb3af2 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:30.874830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae7f8f5-d81b-4cff-b562-37cc61fb3af2 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:30.875101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae7f8f5-d81b-4cff-b562-37cc61fb3af2 None admin] PATH_INFO: `/v3/users/6a1eb60c46aa4d27b91657cd7f9fa60f` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:30.877502 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae7f8f5-d81b-4cff-b562-37cc61fb3af2 None admin] RBAC: Authorizing `identity:get_user(user_id=6a1eb60c46aa4d27b91657cd7f9fa60f)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:30.879186 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae7f8f5-d81b-4cff-b562-37cc61fb3af2 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:30.883795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 88/174] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:30 2022] GET /identity/v3/users/6a1eb60c46aa4d27b91657cd7f9fa60f => generated 275 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:30.884638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae7f8f5-d81b-4cff-b562-37cc61fb3af2 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:30.931733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:30.948549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e03f93-25cb-4820-88da-1222b9a550b4 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:30.991325 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9c6a8f-492d-4211-94f8-dcf0e2f38c60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:30.992337 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9c6a8f-492d-4211-94f8-dcf0e2f38c60 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:30.992819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9c6a8f-492d-4211-94f8-dcf0e2f38c60 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:30.993335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9c6a8f-492d-4211-94f8-dcf0e2f38c60 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:30.994209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9c6a8f-492d-4211-94f8-dcf0e2f38c60 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:30.994825 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9c6a8f-492d-4211-94f8-dcf0e2f38c60 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:30.996322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9c6a8f-492d-4211-94f8-dcf0e2f38c60 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:31.012355 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 87/175] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:12:30 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:31.014181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff04af14-2200-43f8-942e-d9ddb3f7cfa1 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:31.014466 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff04af14-2200-43f8-942e-d9ddb3f7cfa1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:31.014741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff04af14-2200-43f8-942e-d9ddb3f7cfa1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:31.028760 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae7f8f5-d81b-4cff-b562-37cc61fb3af2 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:31.034426 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ff04af14-2200-43f8-942e-d9ddb3f7cfa1 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:31.035621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ff04af14-2200-43f8-942e-d9ddb3f7cfa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:31.038798 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:31.064022 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0165f4-2a82-4990-8883-c1e13a1bb220 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:31.064627 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0165f4-2a82-4990-8883-c1e13a1bb220 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:31.064907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0165f4-2a82-4990-8883-c1e13a1bb220 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:31.065212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0165f4-2a82-4990-8883-c1e13a1bb220 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:31.065939 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0165f4-2a82-4990-8883-c1e13a1bb220 None admin] RBAC: Adding query filter params (role.id=d17954e7ce544f0d90a512340232c60e, scope.project.id=26661cf5060b4b939d8ab02f9ea45daf, user.id=a85a3472aa4a4c95a2bc16b6450e8dcd) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:31.066243 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0165f4-2a82-4990-8883-c1e13a1bb220 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:31.067678 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0165f4-2a82-4990-8883-c1e13a1bb220 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:31.073516 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 89/176] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:12:30 2022] GET /identity/v3/role_assignments?user.id=a85a3472aa4a4c95a2bc16b6450e8dcd&scope.project.id=26661cf5060b4b939d8ab02f9ea45daf&role.id=d17954e7ce544f0d90a512340232c60e => generated 608 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:31.075414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0165f4-2a82-4990-8883-c1e13a1bb220 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:31.114722 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 88/177] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:31.114722 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fad0ea-3967-4100-874b-e4b597809830 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:31.114722 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fad0ea-3967-4100-874b-e4b597809830 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:31.114722 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fad0ea-3967-4100-874b-e4b597809830 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:31.114722 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 89/178] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:31.114722 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6654781a-0e07-4660-bfe3-0e9a5d31d372 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:31.114722 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6654781a-0e07-4660-bfe3-0e9a5d31d372 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:31.116953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6654781a-0e07-4660-bfe3-0e9a5d31d372 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:31.116953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 90/179] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:31.116953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6654781a-0e07-4660-bfe3-0e9a5d31d372 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:31.221814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0165f4-2a82-4990-8883-c1e13a1bb220 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:31.235439 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:31.268801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6654781a-0e07-4660-bfe3-0e9a5d31d372 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:31.285892 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fffdd1-a1f9-4fb0-81f6-1e38b753529a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:31.287128 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fffdd1-a1f9-4fb0-81f6-1e38b753529a None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:31.287622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fffdd1-a1f9-4fb0-81f6-1e38b753529a None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:31.288104 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fffdd1-a1f9-4fb0-81f6-1e38b753529a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:31.310073 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f4af9b-6045-480c-a584-f8ae7cba128b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:31.310714 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f4af9b-6045-480c-a584-f8ae7cba128b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:31.310994 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f4af9b-6045-480c-a584-f8ae7cba128b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:31.311271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f4af9b-6045-480c-a584-f8ae7cba128b None admin] PATH_INFO: `/v3/users/6a1eb60c46aa4d27b91657cd7f9fa60f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:31.312395 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f4af9b-6045-480c-a584-f8ae7cba128b None admin] RBAC: Authorizing `identity:get_user(user_id=6a1eb60c46aa4d27b91657cd7f9fa60f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:31.313871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f4af9b-6045-480c-a584-f8ae7cba128b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:31.315527 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 91/180] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:31 2022] GET /identity/v3/users/6a1eb60c46aa4d27b91657cd7f9fa60f => generated 275 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:31.316212 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f4af9b-6045-480c-a584-f8ae7cba128b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:31.346085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:31.375926 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fffdd1-a1f9-4fb0-81f6-1e38b753529a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:31.377464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fffdd1-a1f9-4fb0-81f6-1e38b753529a None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:31.423456 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f4af9b-6045-480c-a584-f8ae7cba128b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:31.457096 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cfcad2-b5d5-4cd0-9997-b619ca7b1be8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:31.457730 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cfcad2-b5d5-4cd0-9997-b619ca7b1be8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:31.458013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cfcad2-b5d5-4cd0-9997-b619ca7b1be8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:31.458284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cfcad2-b5d5-4cd0-9997-b619ca7b1be8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:31.468179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20cfcad2-b5d5-4cd0-9997-b619ca7b1be8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:31.473108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-20cfcad2-b5d5-4cd0-9997-b619ca7b1be8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:31.473108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:31.473108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:31.473108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:31.473108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:31.473108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:31.473108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:31.473108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:31.473108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:31.473108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:31.473108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:31.473108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:31.473108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:31.473108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:31.473108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:31.473108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:31.474131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:31.474131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:31.474131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:31.474131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:31.474131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:31.474131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:31.474131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:31.474131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:31.474131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:31.474131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:31.474131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:31.474131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:31.474131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:31.474131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:31.474131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:31.474131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:31.474944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:31.474944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:31.474944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:31.474944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:31.474944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:31.474944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:31.474944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:31.474944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:31.474944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:31.474944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:31.474944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:31.474944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:31.474944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 92/181] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:12:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:31.475749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491bd559-24e4-4fe7-a103-ef64557aa4e5 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:31.476301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491bd559-24e4-4fe7-a103-ef64557aa4e5 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:31.476301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491bd559-24e4-4fe7-a103-ef64557aa4e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:31.487030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 90/182] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:12:31.487030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dce64b-54a9-4a0c-93d3-ef37586ee81b None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:31.487030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dce64b-54a9-4a0c-93d3-ef37586ee81b None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:31.487030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dce64b-54a9-4a0c-93d3-ef37586ee81b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:31.498378 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-491bd559-24e4-4fe7-a103-ef64557aa4e5 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:31.499579 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-491bd559-24e4-4fe7-a103-ef64557aa4e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:31.501559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:31.501859 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-42dce64b-54a9-4a0c-93d3-ef37586ee81b None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:31.503101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-42dce64b-54a9-4a0c-93d3-ef37586ee81b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:31.569573 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 93/183] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:31.570041 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 91/184] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:31.571017 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42dce64b-54a9-4a0c-93d3-ef37586ee81b None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:31.573352 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:31.577958 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491bd559-24e4-4fe7-a103-ef64557aa4e5 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:31.700106 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491bd559-24e4-4fe7-a103-ef64557aa4e5 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:31.702127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42dce64b-54a9-4a0c-93d3-ef37586ee81b None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:31.738030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc33c36e-093a-4321-b68c-c4daba137f18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:31.738655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc33c36e-093a-4321-b68c-c4daba137f18 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:31.738998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc33c36e-093a-4321-b68c-c4daba137f18 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:31.739336 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc33c36e-093a-4321-b68c-c4daba137f18 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:31.746940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5512b3-f864-4874-8bfe-0f084ebd821b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:31.747760 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5512b3-f864-4874-8bfe-0f084ebd821b None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:31.748200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5512b3-f864-4874-8bfe-0f084ebd821b None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:31.748621 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5512b3-f864-4874-8bfe-0f084ebd821b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:31.749376 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:31.763869 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff5512b3-f864-4874-8bfe-0f084ebd821b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:31.770682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-ff5512b3-f864-4874-8bfe-0f084ebd821b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:12:31.770682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:31.770682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:31.770682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:31.770682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:31.770682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:31.770682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:31.770682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:31.770682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:31.770682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:31.770682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:31.770682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:31.770682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:31.770682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:31.770682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:31.770682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:31.771777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:31.771777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:31.771777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:31.771777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:31.771777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:31.771777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:31.771777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:31.771777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:31.771777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:31.771777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:31.771777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:31.771777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:31.771777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:31.771777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:31.771777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:31.771777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:31.772739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:31.772739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:31.772739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:31.772739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:31.772739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:31.772739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:31.772739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:31.772739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:31.772739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:31.772739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:31.772739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:12:31.772739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:31.775270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 92/185] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:12:31 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:31.776192 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5512b3-f864-4874-8bfe-0f084ebd821b None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:31.831440 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc33c36e-093a-4321-b68c-c4daba137f18 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:31.833308 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc33c36e-093a-4321-b68c-c4daba137f18 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:31.856183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:31.894297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5512b3-f864-4874-8bfe-0f084ebd821b None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:31.929028 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e7af24-770f-48c8-8c2a-e094e1120e45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:31.929661 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e7af24-770f-48c8-8c2a-e094e1120e45 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:31.929939 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e7af24-770f-48c8-8c2a-e094e1120e45 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:31.930209 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e7af24-770f-48c8-8c2a-e094e1120e45 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:31.930872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e7af24-770f-48c8-8c2a-e094e1120e45 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:31.931177 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e7af24-770f-48c8-8c2a-e094e1120e45 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:31.932547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e7af24-770f-48c8-8c2a-e094e1120e45 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:31.934677 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 94/186] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:12:31.935535 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc33c36e-093a-4321-b68c-c4daba137f18 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:31.942636 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 93/187] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:31.943478 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e7af24-770f-48c8-8c2a-e094e1120e45 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:31.979544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:32.039007 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc33c36e-093a-4321-b68c-c4daba137f18 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:32.041973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e7af24-770f-48c8-8c2a-e094e1120e45 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:32.065098 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:32.072785 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcd87fb-02e9-4f72-a43d-d27bb8a20ecc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:32.073450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcd87fb-02e9-4f72-a43d-d27bb8a20ecc None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:32.073730 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcd87fb-02e9-4f72-a43d-d27bb8a20ecc None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:32.074002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcd87fb-02e9-4f72-a43d-d27bb8a20ecc None admin] PATH_INFO: `/v3/projects/c20f0dd259eb4d55856b1f0848964837` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:32.075147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcd87fb-02e9-4f72-a43d-d27bb8a20ecc None admin] RBAC: Authorizing `identity:get_project(project_id=c20f0dd259eb4d55856b1f0848964837)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:32.075858 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8547fa3-f3fb-4e5e-942c-0d32098a0870 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:32.076458 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8547fa3-f3fb-4e5e-942c-0d32098a0870 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:32.076608 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcd87fb-02e9-4f72-a43d-d27bb8a20ecc None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:32.076738 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8547fa3-f3fb-4e5e-942c-0d32098a0870 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:32.077003 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8547fa3-f3fb-4e5e-942c-0d32098a0870 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:32.077702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8547fa3-f3fb-4e5e-942c-0d32098a0870 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:32.077986 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8547fa3-f3fb-4e5e-942c-0d32098a0870 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:32.078281 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 95/188] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:31 2022] GET /identity/v3/projects/c20f0dd259eb4d55856b1f0848964837 => generated 312 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:32.079049 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcd87fb-02e9-4f72-a43d-d27bb8a20ecc None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:32.079375 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8547fa3-f3fb-4e5e-942c-0d32098a0870 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:32.091826 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 94/189] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:12:31 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:32.091826 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8547fa3-f3fb-4e5e-942c-0d32098a0870 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:32.184741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcd87fb-02e9-4f72-a43d-d27bb8a20ecc None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:32.189852 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8547fa3-f3fb-4e5e-942c-0d32098a0870 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:32.192532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:32.224078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde05d27-d31e-464c-85ab-be9180cd08a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:32.224262 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da9f6e6-a962-4e41-934e-c4a4c2b3d079 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:32.224738 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde05d27-d31e-464c-85ab-be9180cd08a1 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:32.224929 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da9f6e6-a962-4e41-934e-c4a4c2b3d079 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:32.225029 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde05d27-d31e-464c-85ab-be9180cd08a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:32.225241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da9f6e6-a962-4e41-934e-c4a4c2b3d079 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:32.225335 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde05d27-d31e-464c-85ab-be9180cd08a1 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:32.225513 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da9f6e6-a962-4e41-934e-c4a4c2b3d079 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:32.226626 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da9f6e6-a962-4e41-934e-c4a4c2b3d079 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:32.228087 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da9f6e6-a962-4e41-934e-c4a4c2b3d079 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:32.229896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 96/190] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:12:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:32.230445 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da9f6e6-a962-4e41-934e-c4a4c2b3d079 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:32.239512 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde05d27-d31e-464c-85ab-be9180cd08a1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:32.246037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-bde05d27-d31e-464c-85ab-be9180cd08a1 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:12:32.246037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:32.246037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:32.246037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:32.246037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:32.246037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:32.246037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:32.246037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:32.246037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:32.246037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:32.246037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:32.246037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:32.246037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:32.246037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:32.246037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:32.246037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:32.246980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:32.246980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:32.246980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:32.246980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:32.246980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:32.246980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:32.246980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:32.246980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:32.246980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:32.246980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:32.246980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:32.246980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:32.246980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:32.246980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:32.246980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:32.246980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:32.247959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:32.247959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:32.247959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:32.247959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:32.247959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:32.247959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:32.247959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:32.247959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:32.247959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:32.247959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:32.247959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:12:32.247959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:32.247959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 95/191] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:32 2022] GET /identity/v3/roles/reader => generated 84 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:32.247959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde05d27-d31e-464c-85ab-be9180cd08a1 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:32.302126 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:32.338538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da9f6e6-a962-4e41-934e-c4a4c2b3d079 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:32.355927 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde05d27-d31e-464c-85ab-be9180cd08a1 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:32.372503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8684ab51-255a-4c1e-b215-7b0e9ad24bb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:32.373160 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8684ab51-255a-4c1e-b215-7b0e9ad24bb8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:32.373441 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8684ab51-255a-4c1e-b215-7b0e9ad24bb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:32.373712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8684ab51-255a-4c1e-b215-7b0e9ad24bb8 None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:32.374831 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8684ab51-255a-4c1e-b215-7b0e9ad24bb8 None admin] RBAC: Authorizing `identity:get_project(project_id=26661cf5060b4b939d8ab02f9ea45daf)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:32.376286 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8684ab51-255a-4c1e-b215-7b0e9ad24bb8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:32.377958 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 97/192] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:32 2022] GET /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf => generated 298 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:32.378720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8684ab51-255a-4c1e-b215-7b0e9ad24bb8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:32.390062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549caf9b-9f7d-442c-8b76-ba13b4b76abb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:32.390698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549caf9b-9f7d-442c-8b76-ba13b4b76abb None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:32.390981 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549caf9b-9f7d-442c-8b76-ba13b4b76abb None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:32.391256 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549caf9b-9f7d-442c-8b76-ba13b4b76abb None admin] PATH_INFO: `/v3/users/30b88efcb0da48dc8df8edbeb6af20d7` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:32.405006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549caf9b-9f7d-442c-8b76-ba13b4b76abb None admin] RBAC: Authorizing `identity:get_user(user_id=30b88efcb0da48dc8df8edbeb6af20d7)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:32.406543 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549caf9b-9f7d-442c-8b76-ba13b4b76abb None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:32.408146 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 96/193] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:32 2022] GET /identity/v3/users/30b88efcb0da48dc8df8edbeb6af20d7 => generated 283 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:32.409864 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549caf9b-9f7d-442c-8b76-ba13b4b76abb None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:32.426597 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:32.482798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8684ab51-255a-4c1e-b215-7b0e9ad24bb8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:32.508470 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549caf9b-9f7d-442c-8b76-ba13b4b76abb None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:32.511241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:32.517154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd62234-999e-4e69-b292-5fb9f0b5356d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:32.517759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd62234-999e-4e69-b292-5fb9f0b5356d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:32.518039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd62234-999e-4e69-b292-5fb9f0b5356d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:32.518519 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd62234-999e-4e69-b292-5fb9f0b5356d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:32.519025 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd62234-999e-4e69-b292-5fb9f0b5356d None admin] RBAC: Adding query filter params (role.id=ae0cef8120cf4dfc899c30803243a851, scope.project.id=c20f0dd259eb4d55856b1f0848964837, user.id=6a1eb60c46aa4d27b91657cd7f9fa60f) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:32.519335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd62234-999e-4e69-b292-5fb9f0b5356d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:32.520771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd62234-999e-4e69-b292-5fb9f0b5356d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:32.527148 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 98/194] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:12:32 2022] GET /identity/v3/role_assignments?user.id=6a1eb60c46aa4d27b91657cd7f9fa60f&scope.project.id=c20f0dd259eb4d55856b1f0848964837&role.id=ae0cef8120cf4dfc899c30803243a851 => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:32.528084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd62234-999e-4e69-b292-5fb9f0b5356d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:32.547361 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212ef0d9-af2c-4308-909a-f9ef72384fe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:32.548018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212ef0d9-af2c-4308-909a-f9ef72384fe5 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:32.548354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212ef0d9-af2c-4308-909a-f9ef72384fe5 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:32.548570 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212ef0d9-af2c-4308-909a-f9ef72384fe5 None admin] PATH_INFO: `/v3/users/30b88efcb0da48dc8df8edbeb6af20d7` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:32.549756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212ef0d9-af2c-4308-909a-f9ef72384fe5 None admin] RBAC: Authorizing `identity:get_user(user_id=30b88efcb0da48dc8df8edbeb6af20d7)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:32.551371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212ef0d9-af2c-4308-909a-f9ef72384fe5 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:32.553221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 97/195] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:32 2022] GET /identity/v3/users/30b88efcb0da48dc8df8edbeb6af20d7 => generated 283 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:32.553978 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212ef0d9-af2c-4308-909a-f9ef72384fe5 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:32.647642 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd62234-999e-4e69-b292-5fb9f0b5356d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:32.659161 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:32.680614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212ef0d9-af2c-4308-909a-f9ef72384fe5 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:32.698715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74360f9a-afa6-4dcf-a8a3-3c34ad263e67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:32.699452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74360f9a-afa6-4dcf-a8a3-3c34ad263e67 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:32.699731 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74360f9a-afa6-4dcf-a8a3-3c34ad263e67 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:32.700001 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74360f9a-afa6-4dcf-a8a3-3c34ad263e67 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:32.701140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74360f9a-afa6-4dcf-a8a3-3c34ad263e67 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:32.702714 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74360f9a-afa6-4dcf-a8a3-3c34ad263e67 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:32.722611 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956b3d98-e363-4e26-87c2-d35353d843f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:32.723369 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956b3d98-e363-4e26-87c2-d35353d843f8 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:32.723703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956b3d98-e363-4e26-87c2-d35353d843f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:32.723920 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956b3d98-e363-4e26-87c2-d35353d843f8 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:32.724627 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-956b3d98-e363-4e26-87c2-d35353d843f8 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:32.724931 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-956b3d98-e363-4e26-87c2-d35353d843f8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:32.727657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-956b3d98-e363-4e26-87c2-d35353d843f8 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:32.727782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 99/196] 10.222.0.85 () {60 vars in 1132 bytes} [Tue Aug 30 04:12:32 2022] POST /identity/v3/users => generated 297 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:12:32.728227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74360f9a-afa6-4dcf-a8a3-3c34ad263e67 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:32.742103 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 98/197] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:32 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:32.742103 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956b3d98-e363-4e26-87c2-d35353d843f8 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:32.791423 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:32.855729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74360f9a-afa6-4dcf-a8a3-3c34ad263e67 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:32.869002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956b3d98-e363-4e26-87c2-d35353d843f8 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:32.900752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4493e6f-47a9-409a-b7e7-3be1b94df2e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:32.900980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4493e6f-47a9-409a-b7e7-3be1b94df2e0 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:32.901239 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4493e6f-47a9-409a-b7e7-3be1b94df2e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:32.901506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4493e6f-47a9-409a-b7e7-3be1b94df2e0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:32.902236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4493e6f-47a9-409a-b7e7-3be1b94df2e0 None admin] RBAC: Adding query filter params (role.id=a9e862ac528245ca8363bccc5bf35287, scope.project.id=26661cf5060b4b939d8ab02f9ea45daf, user.id=6a1eb60c46aa4d27b91657cd7f9fa60f) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:32.902541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4493e6f-47a9-409a-b7e7-3be1b94df2e0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:32.903990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4493e6f-47a9-409a-b7e7-3be1b94df2e0 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:32.909718 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f8abbd-f0bd-4ddb-b9ba-746f74dfcefb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:32.910343 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f8abbd-f0bd-4ddb-b9ba-746f74dfcefb None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:32.910619 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f8abbd-f0bd-4ddb-b9ba-746f74dfcefb None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:32.911248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f8abbd-f0bd-4ddb-b9ba-746f74dfcefb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:32.911539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 100/198] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:12:32 2022] GET /identity/v3/role_assignments?user.id=6a1eb60c46aa4d27b91657cd7f9fa60f&scope.project.id=26661cf5060b4b939d8ab02f9ea45daf&role.id=a9e862ac528245ca8363bccc5bf35287 => generated 608 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:32.912022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4493e6f-47a9-409a-b7e7-3be1b94df2e0 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:32.945198 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:33.018313 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f8abbd-f0bd-4ddb-b9ba-746f74dfcefb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:33.019956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f8abbd-f0bd-4ddb-b9ba-746f74dfcefb None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:33.062902 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4493e6f-47a9-409a-b7e7-3be1b94df2e0 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:33.064065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:33.111684 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37631cec-d743-4142-ba76-9d623f515d77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:33.112317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37631cec-d743-4142-ba76-9d623f515d77 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:33.112601 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37631cec-d743-4142-ba76-9d623f515d77 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:33.112877 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37631cec-d743-4142-ba76-9d623f515d77 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:33.164917 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 99/199] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:12:33.166234 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f8abbd-f0bd-4ddb-b9ba-746f74dfcefb None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:33.209427 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37631cec-d743-4142-ba76-9d623f515d77 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:33.210926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37631cec-d743-4142-ba76-9d623f515d77 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:33.248943 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:33.302395 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f8abbd-f0bd-4ddb-b9ba-746f74dfcefb None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:33.337843 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 101/200] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:12:33.338693 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37631cec-d743-4142-ba76-9d623f515d77 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:33.350894 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280885ee-fd3e-4bc5-ad8a-de989c4f66f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:33.352564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280885ee-fd3e-4bc5-ad8a-de989c4f66f4 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:33.353069 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280885ee-fd3e-4bc5-ad8a-de989c4f66f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:33.353565 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280885ee-fd3e-4bc5-ad8a-de989c4f66f4 None admin] PATH_INFO: `/v3/users/3124ffc09dfd4a52880634dbc9fcad43` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:33.355753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280885ee-fd3e-4bc5-ad8a-de989c4f66f4 None admin] RBAC: Authorizing `identity:get_user(user_id=3124ffc09dfd4a52880634dbc9fcad43)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:33.357991 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280885ee-fd3e-4bc5-ad8a-de989c4f66f4 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:33.360572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 100/201] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:33 2022] GET /identity/v3/users/3124ffc09dfd4a52880634dbc9fcad43 => generated 289 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:33.362197 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280885ee-fd3e-4bc5-ad8a-de989c4f66f4 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:33.366097 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:33.469283 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37631cec-d743-4142-ba76-9d623f515d77 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:33.474846 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280885ee-fd3e-4bc5-ad8a-de989c4f66f4 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:33.499162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:33.510951 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe69615-c5fd-4afc-b282-521cb1d360d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:33.511545 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0070a6-ae1f-49a9-a247-68aeb35f5339 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:33.512140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe69615-c5fd-4afc-b282-521cb1d360d2 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:33.512247 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0070a6-ae1f-49a9-a247-68aeb35f5339 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:33.512444 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0070a6-ae1f-49a9-a247-68aeb35f5339 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:33.512716 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0070a6-ae1f-49a9-a247-68aeb35f5339 None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:33.513143 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe69615-c5fd-4afc-b282-521cb1d360d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:33.513614 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe69615-c5fd-4afc-b282-521cb1d360d2 None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:33.513885 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0070a6-ae1f-49a9-a247-68aeb35f5339 None admin] RBAC: Authorizing `identity:get_project(project_id=008c6a0a4a114009bc7c692aabd0b0c8)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:33.515199 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe69615-c5fd-4afc-b282-521cb1d360d2 None admin] RBAC: Authorizing `identity:get_project(project_id=008c6a0a4a114009bc7c692aabd0b0c8)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:33.515459 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0070a6-ae1f-49a9-a247-68aeb35f5339 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:33.517083 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe69615-c5fd-4afc-b282-521cb1d360d2 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:33.517494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 101/202] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:33 2022] GET /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8 => generated 302 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:33.518459 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0070a6-ae1f-49a9-a247-68aeb35f5339 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:33.519581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 102/203] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:33 2022] GET /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8 => generated 302 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:33.525491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe69615-c5fd-4afc-b282-521cb1d360d2 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:33.611458 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:33.641906 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe69615-c5fd-4afc-b282-521cb1d360d2 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:33.656365 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0070a6-ae1f-49a9-a247-68aeb35f5339 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:33.685751 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcef942-181f-456a-a0af-d7cbea971668 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:33.686918 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcef942-181f-456a-a0af-d7cbea971668 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:33.687408 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcef942-181f-456a-a0af-d7cbea971668 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:33.688092 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcef942-181f-456a-a0af-d7cbea971668 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:33.689686 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcef942-181f-456a-a0af-d7cbea971668 None admin] RBAC: Adding query filter params (role.id=a9e862ac528245ca8363bccc5bf35287, scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8, user.id=30b88efcb0da48dc8df8edbeb6af20d7) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:33.690199 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcef942-181f-456a-a0af-d7cbea971668 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:33.691833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcef942-181f-456a-a0af-d7cbea971668 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:33.694192 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f4bfef-b331-47ff-be6c-cebbb81372b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:33.694863 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f4bfef-b331-47ff-be6c-cebbb81372b7 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:33.695205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f4bfef-b331-47ff-be6c-cebbb81372b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:33.695534 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f4bfef-b331-47ff-be6c-cebbb81372b7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:33.705726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 103/204] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:12:33 2022] GET /identity/v3/role_assignments?user.id=30b88efcb0da48dc8df8edbeb6af20d7&scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8&role.id=a9e862ac528245ca8363bccc5bf35287 => generated 259 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:33.706234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcef942-181f-456a-a0af-d7cbea971668 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:33.828955 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:33.829165 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f4bfef-b331-47ff-be6c-cebbb81372b7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:33.830622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f4bfef-b331-47ff-be6c-cebbb81372b7 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:33.908897 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:33.928214 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcef942-181f-456a-a0af-d7cbea971668 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:33.955131 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 102/205] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 437 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:12:33.961593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f4bfef-b331-47ff-be6c-cebbb81372b7 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:33.998050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8678d5-5326-4254-90d4-24b3d4535efa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:33.998777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8678d5-5326-4254-90d4-24b3d4535efa None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:33.999003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8678d5-5326-4254-90d4-24b3d4535efa None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:33.999211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8678d5-5326-4254-90d4-24b3d4535efa None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:33.999946 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8678d5-5326-4254-90d4-24b3d4535efa None admin] RBAC: Adding query filter params (role.id=d17954e7ce544f0d90a512340232c60e, scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8, user.id=30b88efcb0da48dc8df8edbeb6af20d7) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:34.000249 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8678d5-5326-4254-90d4-24b3d4535efa None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:34.001735 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8678d5-5326-4254-90d4-24b3d4535efa None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:34.008619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 104/206] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:12:33 2022] GET /identity/v3/role_assignments?user.id=30b88efcb0da48dc8df8edbeb6af20d7&scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8&role.id=d17954e7ce544f0d90a512340232c60e => generated 259 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:34.083169 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f4bfef-b331-47ff-be6c-cebbb81372b7 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:34.137833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a9c12e-7e81-49b5-82c9-5619e1be73c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:34.138864 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a9c12e-7e81-49b5-82c9-5619e1be73c2 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:34.139356 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a9c12e-7e81-49b5-82c9-5619e1be73c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:34.139850 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a9c12e-7e81-49b5-82c9-5619e1be73c2 None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:34.141451 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a9c12e-7e81-49b5-82c9-5619e1be73c2 None admin] RBAC: Authorizing `identity:get_project(project_id=26661cf5060b4b939d8ab02f9ea45daf)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:34.143276 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a9c12e-7e81-49b5-82c9-5619e1be73c2 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:34.145811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 103/207] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:33 2022] GET /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf => generated 298 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:34.152311 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8678d5-5326-4254-90d4-24b3d4535efa None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:34.290074 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8678d5-5326-4254-90d4-24b3d4535efa None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:34.323734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:34.338778 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5459b71-aa1d-41d7-b5cc-a6d29dfd844a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:34.339398 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5459b71-aa1d-41d7-b5cc-a6d29dfd844a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:34.339682 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5459b71-aa1d-41d7-b5cc-a6d29dfd844a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:34.339958 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5459b71-aa1d-41d7-b5cc-a6d29dfd844a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:34.340699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5459b71-aa1d-41d7-b5cc-a6d29dfd844a None admin] RBAC: Adding query filter params (role.id=902fd8d9a4ef48058e1b17e0429ede58, scope.project.id=26661cf5060b4b939d8ab02f9ea45daf, user.id=3124ffc09dfd4a52880634dbc9fcad43) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:34.341005 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5459b71-aa1d-41d7-b5cc-a6d29dfd844a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:34.342476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5459b71-aa1d-41d7-b5cc-a6d29dfd844a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:34.354432 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 105/208] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:12:34 2022] GET /identity/v3/role_assignments?user.id=3124ffc09dfd4a52880634dbc9fcad43&scope.project.id=26661cf5060b4b939d8ab02f9ea45daf&role.id=902fd8d9a4ef48058e1b17e0429ede58 => generated 608 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:35.064393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd34448e-2b4a-4c29-b668-b62641d085c4 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:35.064963 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd34448e-2b4a-4c29-b668-b62641d085c4 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:35.064963 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd34448e-2b4a-4c29-b668-b62641d085c4 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:35.066662 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 104/209] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:35.073186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f52489a-1b2d-434c-bfc3-219793090e9b None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:35.073687 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f52489a-1b2d-434c-bfc3-219793090e9b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:35.073758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f52489a-1b2d-434c-bfc3-219793090e9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:35.090668 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f50265c-2ceb-4173-a8a4-1ebbf8526d02 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:35.091298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f50265c-2ceb-4173-a8a4-1ebbf8526d02 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:35.091772 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f50265c-2ceb-4173-a8a4-1ebbf8526d02 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:35.093315 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5f52489a-1b2d-434c-bfc3-219793090e9b None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:35.093568 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 105/210] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:12:35.094521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5f52489a-1b2d-434c-bfc3-219793090e9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:35.098993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db32b89-2b48-4dc3-8161-417df7e011d7 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:35.099275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db32b89-2b48-4dc3-8161-417df7e011d7 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:35.100084 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db32b89-2b48-4dc3-8161-417df7e011d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:35.133953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-2db32b89-2b48-4dc3-8161-417df7e011d7 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:35.135120 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-2db32b89-2b48-4dc3-8161-417df7e011d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:35.178887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:35.183986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 106/211] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:35.224305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 106/212] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:35.456080 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f52489a-1b2d-434c-bfc3-219793090e9b None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:35.492653 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db32b89-2b48-4dc3-8161-417df7e011d7 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:35.563870 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f52489a-1b2d-434c-bfc3-219793090e9b None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:35.581810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:35.598292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db32b89-2b48-4dc3-8161-417df7e011d7 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:35.603005 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622270cc-0073-4e9a-85f5-bb3dda5299b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:35.603885 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622270cc-0073-4e9a-85f5-bb3dda5299b5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:35.604361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622270cc-0073-4e9a-85f5-bb3dda5299b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:35.604857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622270cc-0073-4e9a-85f5-bb3dda5299b5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:35.615931 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622270cc-0073-4e9a-85f5-bb3dda5299b5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:35.621107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-622270cc-0073-4e9a-85f5-bb3dda5299b5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:35.621107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:35.621107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:35.621107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:35.621107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:35.621107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:35.621107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:35.621107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:35.621107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:35.621107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:35.621107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:35.621107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:35.621107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:35.621107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:35.621107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:35.621107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:35.621975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:35.621975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:35.621975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:35.621975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:35.621975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:35.621975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:35.621975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:35.621975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:35.621975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:35.621975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:35.621975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:35.621975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:35.621975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:35.621975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:35.621975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:35.621975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:35.622750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:35.622750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:35.622750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:35.622750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:35.622750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:35.622750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:35.622750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:35.622750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:35.622750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:35.622750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:35.622750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:35.622750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:35.625338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 107/213] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:35 2022] GET /identity/v3/roles/member => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:35.632192 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622270cc-0073-4e9a-85f5-bb3dda5299b5 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:35.636702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac8daac-3a3f-4ae3-a888-0a1db6476362 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:35.637357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac8daac-3a3f-4ae3-a888-0a1db6476362 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:35.637632 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac8daac-3a3f-4ae3-a888-0a1db6476362 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:35.637899 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac8daac-3a3f-4ae3-a888-0a1db6476362 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:35.639774 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac8daac-3a3f-4ae3-a888-0a1db6476362 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:35.640420 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac8daac-3a3f-4ae3-a888-0a1db6476362 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:35.642255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 107/214] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:12:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:35.648041 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac8daac-3a3f-4ae3-a888-0a1db6476362 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:35.684177 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:35.760177 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac8daac-3a3f-4ae3-a888-0a1db6476362 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:35.767783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622270cc-0073-4e9a-85f5-bb3dda5299b5 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:35.804554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b3aedb-80b8-4b0a-857f-c5fb3b6e93b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:35.805159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b3aedb-80b8-4b0a-857f-c5fb3b6e93b4 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:35.805455 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b3aedb-80b8-4b0a-857f-c5fb3b6e93b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:35.805712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b3aedb-80b8-4b0a-857f-c5fb3b6e93b4 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:35.806374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b3aedb-80b8-4b0a-857f-c5fb3b6e93b4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:35.806653 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b3aedb-80b8-4b0a-857f-c5fb3b6e93b4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:35.808048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b3aedb-80b8-4b0a-857f-c5fb3b6e93b4 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:35.809735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdc595b-31aa-42c2-a791-463b9ee5b914 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:35.811020 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdc595b-31aa-42c2-a791-463b9ee5b914 None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:35.811498 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdc595b-31aa-42c2-a791-463b9ee5b914 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:35.811959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdc595b-31aa-42c2-a791-463b9ee5b914 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:35.812802 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbdc595b-31aa-42c2-a791-463b9ee5b914 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:35.815281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbdc595b-31aa-42c2-a791-463b9ee5b914 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:35.817067 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 108/215] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:35 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:35.822825 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b3aedb-80b8-4b0a-857f-c5fb3b6e93b4 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:35.845502 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 108/216] 10.222.0.85 () {60 vars in 1132 bytes} [Tue Aug 30 04:12:35 2022] POST /identity/v3/users => generated 297 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:12:35.876074 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:35.928580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79957f10-af61-4937-9545-b70ecea57f70 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:35.928923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79957f10-af61-4937-9545-b70ecea57f70 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:35.929166 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79957f10-af61-4937-9545-b70ecea57f70 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:35.934666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 109/217] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:12:35.936499 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9618235-1d34-4d58-ae6f-1d64fe6fce45 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:35.936778 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9618235-1d34-4d58-ae6f-1d64fe6fce45 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:35.937065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9618235-1d34-4d58-ae6f-1d64fe6fce45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:35.955308 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b3aedb-80b8-4b0a-857f-c5fb3b6e93b4 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:35.956620 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-d9618235-1d34-4d58-ae6f-1d64fe6fce45 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:35.958064 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d9618235-1d34-4d58-ae6f-1d64fe6fce45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:35.998788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236b3c9f-2c52-4120-a8ea-77af3a4abd84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:35.999789 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236b3c9f-2c52-4120-a8ea-77af3a4abd84 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:36.000298 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236b3c9f-2c52-4120-a8ea-77af3a4abd84 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:36.000769 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236b3c9f-2c52-4120-a8ea-77af3a4abd84 None admin] PATH_INFO: `/v3/users/0fba3b2cdd9843f0bf62fe574d3e3ec1` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:36.018102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236b3c9f-2c52-4120-a8ea-77af3a4abd84 None admin] RBAC: Authorizing `identity:get_user(user_id=0fba3b2cdd9843f0bf62fe574d3e3ec1)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:36.021985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236b3c9f-2c52-4120-a8ea-77af3a4abd84 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:36.025981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 109/218] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:35 2022] GET /identity/v3/users/0fba3b2cdd9843f0bf62fe574d3e3ec1 => generated 297 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:36.033474 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236b3c9f-2c52-4120-a8ea-77af3a4abd84 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:36.049122 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 110/219] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:36.142137 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:36.149538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236b3c9f-2c52-4120-a8ea-77af3a4abd84 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:36.156832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74abbce-1141-4aca-8615-9ecaab03e4d7 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:36.157368 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74abbce-1141-4aca-8615-9ecaab03e4d7 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:36.157368 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74abbce-1141-4aca-8615-9ecaab03e4d7 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:36.159755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 111/220] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:12:36.164482 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f733e653-a501-40f8-8f69-711a11120363 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:36.164764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f733e653-a501-40f8-8f69-711a11120363 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:36.165047 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f733e653-a501-40f8-8f69-711a11120363 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:36.190061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0a8625-a261-4342-b60b-6475d82736d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:36.190262 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-f733e653-a501-40f8-8f69-711a11120363 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:36.190684 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0a8625-a261-4342-b60b-6475d82736d6 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:36.190966 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0a8625-a261-4342-b60b-6475d82736d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:36.191284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0a8625-a261-4342-b60b-6475d82736d6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:36.191400 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f733e653-a501-40f8-8f69-711a11120363 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:36.214444 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:36.278999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0a8625-a261-4342-b60b-6475d82736d6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:36.280393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0a8625-a261-4342-b60b-6475d82736d6 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:36.286155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 112/221] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:36.336590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f733e653-a501-40f8-8f69-711a11120363 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:36.389533 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:36.393170 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 110/222] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:12:36.399563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0a8625-a261-4342-b60b-6475d82736d6 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:36.536803 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f733e653-a501-40f8-8f69-711a11120363 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:36.582350 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0a8625-a261-4342-b60b-6475d82736d6 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:36.604641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d09847-7e20-4e73-b567-451dffe49e41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:36.605356 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d09847-7e20-4e73-b567-451dffe49e41 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:36.605703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d09847-7e20-4e73-b567-451dffe49e41 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:36.605972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d09847-7e20-4e73-b567-451dffe49e41 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:36.617531 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d09847-7e20-4e73-b567-451dffe49e41 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:36.622962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-35d09847-7e20-4e73-b567-451dffe49e41 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:12:36.622962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:36.622962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:36.622962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:36.622962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:36.622962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:36.622962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:36.622962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:36.622962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:36.622962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:36.622962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:36.622962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:36.622962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:36.622962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:36.622962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:36.622962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:36.623828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:36.623828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:36.623828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:36.623828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:36.623828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:36.623828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:36.623828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:36.623828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:36.623828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:36.623828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:36.623828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:36.623828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:36.623828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:36.623828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:36.623828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:36.623828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:36.624600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:36.624600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:36.624600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:36.624600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:36.624600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:36.624600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:36.624600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:36.624600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:36.624600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:36.624600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:36.624600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:12:36.624600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:36.624600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 113/223] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:12:36 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:36.625733 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d09847-7e20-4e73-b567-451dffe49e41 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:36.636251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2142e9-2435-40e5-a475-e9ac1ecf3c96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:36.637017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2142e9-2435-40e5-a475-e9ac1ecf3c96 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:36.637382 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2142e9-2435-40e5-a475-e9ac1ecf3c96 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:36.637810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2142e9-2435-40e5-a475-e9ac1ecf3c96 None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:36.639448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2142e9-2435-40e5-a475-e9ac1ecf3c96 None admin] RBAC: Authorizing `identity:get_project(project_id=008c6a0a4a114009bc7c692aabd0b0c8)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:36.641376 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2142e9-2435-40e5-a475-e9ac1ecf3c96 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:36.643904 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 111/224] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:36 2022] GET /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8 => generated 302 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:36.644824 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2142e9-2435-40e5-a475-e9ac1ecf3c96 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:36.665418 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:36.719206 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:36.737172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d09847-7e20-4e73-b567-451dffe49e41 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:36.761279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2142e9-2435-40e5-a475-e9ac1ecf3c96 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:36.783441 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6961bec7-718e-497f-a456-3e4e4dcd87e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:36.784458 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6961bec7-718e-497f-a456-3e4e4dcd87e0 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:36.784736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6961bec7-718e-497f-a456-3e4e4dcd87e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:36.785016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6961bec7-718e-497f-a456-3e4e4dcd87e0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:36.799254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6961bec7-718e-497f-a456-3e4e4dcd87e0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:36.801429 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155bb0ac-6de0-4589-bd2c-f698a160e06d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:36.802147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155bb0ac-6de0-4589-bd2c-f698a160e06d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:36.803068 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155bb0ac-6de0-4589-bd2c-f698a160e06d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:36.803440 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155bb0ac-6de0-4589-bd2c-f698a160e06d None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:36.804217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155bb0ac-6de0-4589-bd2c-f698a160e06d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:36.804617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155bb0ac-6de0-4589-bd2c-f698a160e06d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:36.804832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-6961bec7-718e-497f-a456-3e4e4dcd87e0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:36.804832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:36.804832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:36.804832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:36.804832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:36.804832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:36.804832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:36.804832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:36.804832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:36.804832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:36.804832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:36.804832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:36.804832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:36.804832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:36.804832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:36.804832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:36.805757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:36.805757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:36.805757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:36.805757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:36.805757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:36.805757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:36.805757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:36.805757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:36.805757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:36.805757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:36.805757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:36.805757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:36.805757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:36.805757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:36.805757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:36.805757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:36.806525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:36.806525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:36.806525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:36.806525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:36.806525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:36.806525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:36.806525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:36.806525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:36.806525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:36.806525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:36.806525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:36.806525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:36.806525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 114/225] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:12:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:36.807165 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155bb0ac-6de0-4589-bd2c-f698a160e06d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:36.807231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6961bec7-718e-497f-a456-3e4e4dcd87e0 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:36.820927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 112/226] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:12:36 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:36.822034 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155bb0ac-6de0-4589-bd2c-f698a160e06d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:36.902527 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:36.929521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155bb0ac-6de0-4589-bd2c-f698a160e06d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:36.943109 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6961bec7-718e-497f-a456-3e4e4dcd87e0 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:36.957717 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:36.968924 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69123bd-652f-45ec-9e2e-3dd1eaaa5ad7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:36.969645 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69123bd-652f-45ec-9e2e-3dd1eaaa5ad7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:36.969988 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69123bd-652f-45ec-9e2e-3dd1eaaa5ad7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:36.970403 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69123bd-652f-45ec-9e2e-3dd1eaaa5ad7 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:36.971196 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69123bd-652f-45ec-9e2e-3dd1eaaa5ad7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:36.971562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69123bd-652f-45ec-9e2e-3dd1eaaa5ad7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:36.973242 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69123bd-652f-45ec-9e2e-3dd1eaaa5ad7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:36.986080 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533d7f15-77e2-4035-bb19-c5fa76c05e36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:36.986687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533d7f15-77e2-4035-bb19-c5fa76c05e36 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:36.986956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533d7f15-77e2-4035-bb19-c5fa76c05e36 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:36.987223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533d7f15-77e2-4035-bb19-c5fa76c05e36 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:36.988228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533d7f15-77e2-4035-bb19-c5fa76c05e36 None admin] RBAC: Adding query filter params (role.id=ae0cef8120cf4dfc899c30803243a851, scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8, user.id=0fba3b2cdd9843f0bf62fe574d3e3ec1) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:36.988412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 113/227] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:36.988558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533d7f15-77e2-4035-bb19-c5fa76c05e36 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:36.988999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69123bd-652f-45ec-9e2e-3dd1eaaa5ad7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:36.990056 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533d7f15-77e2-4035-bb19-c5fa76c05e36 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:37.000189 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 115/228] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:12:36 2022] GET /identity/v3/role_assignments?user.id=0fba3b2cdd9843f0bf62fe574d3e3ec1&scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8&role.id=ae0cef8120cf4dfc899c30803243a851 => generated 259 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:37.001602 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533d7f15-77e2-4035-bb19-c5fa76c05e36 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:37.143043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533d7f15-77e2-4035-bb19-c5fa76c05e36 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:37.168224 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69123bd-652f-45ec-9e2e-3dd1eaaa5ad7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:37.188680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6455fdb4-2789-48b7-9293-6c44e5d60ca8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:37.190125 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6455fdb4-2789-48b7-9293-6c44e5d60ca8 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:37.190582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6455fdb4-2789-48b7-9293-6c44e5d60ca8 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:37.191033 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6455fdb4-2789-48b7-9293-6c44e5d60ca8 None admin] PATH_INFO: `/v3/users/30b88efcb0da48dc8df8edbeb6af20d7` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:37.192528 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6455fdb4-2789-48b7-9293-6c44e5d60ca8 None admin] RBAC: Authorizing `identity:get_user(user_id=30b88efcb0da48dc8df8edbeb6af20d7)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:37.194849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6455fdb4-2789-48b7-9293-6c44e5d60ca8 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:37.196834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 116/229] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:37 2022] GET /identity/v3/users/30b88efcb0da48dc8df8edbeb6af20d7 => generated 283 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:37.202790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6455fdb4-2789-48b7-9293-6c44e5d60ca8 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:37.205562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200df8dc-7997-4cc5-9785-8b66499b25dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:37.206292 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200df8dc-7997-4cc5-9785-8b66499b25dc None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:37.206661 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200df8dc-7997-4cc5-9785-8b66499b25dc None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:37.207226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200df8dc-7997-4cc5-9785-8b66499b25dc None admin] PATH_INFO: `/v3/users/30b88efcb0da48dc8df8edbeb6af20d7` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:37.208031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:37.208893 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200df8dc-7997-4cc5-9785-8b66499b25dc None admin] RBAC: Authorizing `identity:get_user(user_id=30b88efcb0da48dc8df8edbeb6af20d7)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:37.210448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200df8dc-7997-4cc5-9785-8b66499b25dc None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:37.212124 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 114/230] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:36 2022] GET /identity/v3/users/30b88efcb0da48dc8df8edbeb6af20d7 => generated 283 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:37.217765 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200df8dc-7997-4cc5-9785-8b66499b25dc None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:37.278141 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:37.329415 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6455fdb4-2789-48b7-9293-6c44e5d60ca8 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:37.330633 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200df8dc-7997-4cc5-9785-8b66499b25dc None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:37.368203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb8f825-2ecf-474a-9cc6-ca2104529b2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:37.368409 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c0a48f-3ba7-407a-bec4-c72682589b34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:37.368768 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c0a48f-3ba7-407a-bec4-c72682589b34 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:37.369068 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c0a48f-3ba7-407a-bec4-c72682589b34 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:37.369370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c0a48f-3ba7-407a-bec4-c72682589b34 None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:37.369861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb8f825-2ecf-474a-9cc6-ca2104529b2c None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:37.370327 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb8f825-2ecf-474a-9cc6-ca2104529b2c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:37.370490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c0a48f-3ba7-407a-bec4-c72682589b34 None admin] RBAC: Authorizing `identity:get_project(project_id=008c6a0a4a114009bc7c692aabd0b0c8)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:37.370923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb8f825-2ecf-474a-9cc6-ca2104529b2c None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:37.371947 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c0a48f-3ba7-407a-bec4-c72682589b34 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:37.372422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb8f825-2ecf-474a-9cc6-ca2104529b2c None admin] RBAC: Authorizing `identity:get_project(project_id=008c6a0a4a114009bc7c692aabd0b0c8)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:37.373910 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 115/231] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:37 2022] GET /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8 => generated 302 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:37.374343 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb8f825-2ecf-474a-9cc6-ca2104529b2c None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:37.376434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 117/232] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:37 2022] GET /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8 => generated 302 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:37.381078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c0a48f-3ba7-407a-bec4-c72682589b34 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:37.383339 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb8f825-2ecf-474a-9cc6-ca2104529b2c None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:37.473428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:37.497713 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb8f825-2ecf-474a-9cc6-ca2104529b2c None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:37.511782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c0a48f-3ba7-407a-bec4-c72682589b34 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:37.558675 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:37.563024 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1adb3ab-a2df-409a-b0b3-d23d6c40999d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:37.563688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1adb3ab-a2df-409a-b0b3-d23d6c40999d None admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:37.563964 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1adb3ab-a2df-409a-b0b3-d23d6c40999d None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:37.564235 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1adb3ab-a2df-409a-b0b3-d23d6c40999d None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8/users/30b88efcb0da48dc8df8edbeb6af20d7/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:37.567423 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1adb3ab-a2df-409a-b0b3-d23d6c40999d None admin] RBAC: Authorizing `identity:create_grant(project_id=008c6a0a4a114009bc7c692aabd0b0c8, user_id=30b88efcb0da48dc8df8edbeb6af20d7, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:37.571267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1adb3ab-a2df-409a-b0b3-d23d6c40999d None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:37.577389 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89241a2c-523d-45c7-8b29-6aea37a6c02a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:37.578054 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89241a2c-523d-45c7-8b29-6aea37a6c02a None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:37.578326 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89241a2c-523d-45c7-8b29-6aea37a6c02a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:37.578593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89241a2c-523d-45c7-8b29-6aea37a6c02a None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8/users/30b88efcb0da48dc8df8edbeb6af20d7/roles/a9e862ac528245ca8363bccc5bf35287` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:37.581397 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89241a2c-523d-45c7-8b29-6aea37a6c02a None admin] RBAC: Authorizing `identity:create_grant(project_id=008c6a0a4a114009bc7c692aabd0b0c8, user_id=30b88efcb0da48dc8df8edbeb6af20d7, role_id=a9e862ac528245ca8363bccc5bf35287)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:37.583055 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89241a2c-523d-45c7-8b29-6aea37a6c02a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:37.598387 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 118/233] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:12:37 2022] PUT /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8/users/30b88efcb0da48dc8df8edbeb6af20d7/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:12:37.605924 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 116/234] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:12:37 2022] PUT /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8/users/30b88efcb0da48dc8df8edbeb6af20d7/roles/a9e862ac528245ca8363bccc5bf35287 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:12:38.253205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96656b0-cefd-4f78-b90c-c25017a6fd80 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:38.254203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96656b0-cefd-4f78-b90c-c25017a6fd80 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:38.254720 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96656b0-cefd-4f78-b90c-c25017a6fd80 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:38.256683 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 119/235] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:38 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:38.262479 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127a78d7-dab2-4cf4-a0da-a12a02115b6a None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:38.263339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127a78d7-dab2-4cf4-a0da-a12a02115b6a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:38.263827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127a78d7-dab2-4cf4-a0da-a12a02115b6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:38.283563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-127a78d7-dab2-4cf4-a0da-a12a02115b6a None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:38.285276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-127a78d7-dab2-4cf4-a0da-a12a02115b6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:38.358482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 117/236] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:38.420441 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627988a2-e638-42f2-8c96-2551a1c884d6 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:38.420718 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627988a2-e638-42f2-8c96-2551a1c884d6 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:38.420911 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627988a2-e638-42f2-8c96-2551a1c884d6 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:38.422451 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 120/237] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:38.439025 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ceafd4-b606-495a-afb1-4634bc7c02b9 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:38.439607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ceafd4-b606-495a-afb1-4634bc7c02b9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:38.440095 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ceafd4-b606-495a-afb1-4634bc7c02b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:38.463516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-d5ceafd4-b606-495a-afb1-4634bc7c02b9 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:38.465189 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d5ceafd4-b606-495a-afb1-4634bc7c02b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:38.537363 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 118/238] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:38.691096 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627988a2-e638-42f2-8c96-2551a1c884d6 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:38.826603 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627988a2-e638-42f2-8c96-2551a1c884d6 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:38.849756 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ceafd4-b606-495a-afb1-4634bc7c02b9 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:38.861544 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:38.869842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec37c920-079f-47d1-be37-a3d6c39b3ff5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:38.871857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec37c920-079f-47d1-be37-a3d6c39b3ff5 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:38.872131 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec37c920-079f-47d1-be37-a3d6c39b3ff5 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:38.872396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec37c920-079f-47d1-be37-a3d6c39b3ff5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:38.874295 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec37c920-079f-47d1-be37-a3d6c39b3ff5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:38.875749 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec37c920-079f-47d1-be37-a3d6c39b3ff5 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:38.877349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 121/239] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:12:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:38.883735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec37c920-079f-47d1-be37-a3d6c39b3ff5 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:38.953241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:39.053622 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ceafd4-b606-495a-afb1-4634bc7c02b9 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:39.055746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec37c920-079f-47d1-be37-a3d6c39b3ff5 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:39.108076 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c69c08c-81fa-4275-8c76-cd7407bc6383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:39.108744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c69c08c-81fa-4275-8c76-cd7407bc6383 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:39.109025 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c69c08c-81fa-4275-8c76-cd7407bc6383 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:39.109319 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c69c08c-81fa-4275-8c76-cd7407bc6383 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:39.132276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c69c08c-81fa-4275-8c76-cd7407bc6383 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:39.137245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-6c69c08c-81fa-4275-8c76-cd7407bc6383 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:12:39.137245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:39.137245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:39.137245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:39.137245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:39.137245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:39.137245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:39.137245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:39.137245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:39.137245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:39.137245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:39.137245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:39.137245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:39.137245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:39.137245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:39.137245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:39.138113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:39.138113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:39.138113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:39.138113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:39.138113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:39.138113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:39.138113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:39.138113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:39.138113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:39.138113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:39.138113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:39.138113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:39.138113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:39.138113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:39.138113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:39.138113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:39.138879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:39.138879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:39.138879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:39.138879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:39.138879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:39.138879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:39.138879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:39.138879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:39.138879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:39.138879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:39.138879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:12:39.138879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:39.138879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 119/240] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:38 2022] GET /identity/v3/roles/reader => generated 84 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:39.145409 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c69c08c-81fa-4275-8c76-cd7407bc6383 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:39.145533 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c283ac-7119-45c3-a817-e9c6f5bf0473 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:39.146736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c283ac-7119-45c3-a817-e9c6f5bf0473 None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:39.147019 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c283ac-7119-45c3-a817-e9c6f5bf0473 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:39.147287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c283ac-7119-45c3-a817-e9c6f5bf0473 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:39.147939 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c283ac-7119-45c3-a817-e9c6f5bf0473 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:39.149399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c283ac-7119-45c3-a817-e9c6f5bf0473 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:39.195541 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 122/241] 10.222.0.85 () {60 vars in 1132 bytes} [Tue Aug 30 04:12:38 2022] POST /identity/v3/users => generated 293 bytes in 312 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:12:39.289986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c69c08c-81fa-4275-8c76-cd7407bc6383 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:39.292459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:39.342703 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5605c42f-38ac-43fe-b62d-e2b74347a038 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:39.343347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5605c42f-38ac-43fe-b62d-e2b74347a038 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:39.343685 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5605c42f-38ac-43fe-b62d-e2b74347a038 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:39.343966 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5605c42f-38ac-43fe-b62d-e2b74347a038 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:39.344712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5605c42f-38ac-43fe-b62d-e2b74347a038 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:39.345020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5605c42f-38ac-43fe-b62d-e2b74347a038 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:39.346568 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5605c42f-38ac-43fe-b62d-e2b74347a038 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:39.356308 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 120/242] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:39 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:39.362259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c283ac-7119-45c3-a817-e9c6f5bf0473 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:39.485319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:39.503013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501c8746-8f33-42bd-a08d-eeb29fd0d73d None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:39.503378 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501c8746-8f33-42bd-a08d-eeb29fd0d73d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:39.503564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501c8746-8f33-42bd-a08d-eeb29fd0d73d None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:39.505193 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 121/243] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:39.510981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d104066d-2049-4cb6-a122-ecb40505b376 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:39.515805 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d104066d-2049-4cb6-a122-ecb40505b376 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:39.515805 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d104066d-2049-4cb6-a122-ecb40505b376 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:39.546343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-d104066d-2049-4cb6-a122-ecb40505b376 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:39.547497 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d104066d-2049-4cb6-a122-ecb40505b376 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:39.576807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c283ac-7119-45c3-a817-e9c6f5bf0473 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:39.648715 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323f7d33-de68-4299-9704-827ddd1388e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:39.650076 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323f7d33-de68-4299-9704-827ddd1388e6 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:39.650577 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323f7d33-de68-4299-9704-827ddd1388e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:39.651048 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323f7d33-de68-4299-9704-827ddd1388e6 None admin] PATH_INFO: `/v3/users/dba522ffd9424baf80a496f4095561b7` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:39.668533 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 122/244] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:39.672965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323f7d33-de68-4299-9704-827ddd1388e6 None admin] RBAC: Authorizing `identity:get_user(user_id=dba522ffd9424baf80a496f4095561b7)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:39.672965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323f7d33-de68-4299-9704-827ddd1388e6 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:39.672965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 123/245] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:39 2022] GET /identity/v3/users/dba522ffd9424baf80a496f4095561b7 => generated 297 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:39.679856 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d104066d-2049-4cb6-a122-ecb40505b376 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:39.810240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:39.813720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d104066d-2049-4cb6-a122-ecb40505b376 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:39.869717 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9692d33-2536-42e5-98b7-fca59fda6af6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:39.870906 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9692d33-2536-42e5-98b7-fca59fda6af6 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:39.871818 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9692d33-2536-42e5-98b7-fca59fda6af6 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:39.872458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9692d33-2536-42e5-98b7-fca59fda6af6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:39.975635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9692d33-2536-42e5-98b7-fca59fda6af6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:39.977440 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9692d33-2536-42e5-98b7-fca59fda6af6 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:40.039112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323f7d33-de68-4299-9704-827ddd1388e6 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:40.095560 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:40.111800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 123/246] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:12:40.114035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d2bbde-3540-4a21-af12-ff69a336764f None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:40.114324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d2bbde-3540-4a21-af12-ff69a336764f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:40.114598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d2bbde-3540-4a21-af12-ff69a336764f None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:40.117876 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 124/247] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:40.122927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669f8956-d2fb-4da1-bc51-9fdf6ee94b8a None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:40.123389 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669f8956-d2fb-4da1-bc51-9fdf6ee94b8a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:40.123898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669f8956-d2fb-4da1-bc51-9fdf6ee94b8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:40.144680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-669f8956-d2fb-4da1-bc51-9fdf6ee94b8a None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:40.145665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-669f8956-d2fb-4da1-bc51-9fdf6ee94b8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:40.201186 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323f7d33-de68-4299-9704-827ddd1388e6 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:40.216994 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:40.231287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 125/248] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:40.232456 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669f8956-d2fb-4da1-bc51-9fdf6ee94b8a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:40.258857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb399a43-c060-4dc1-bf07-3c76cf919989 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:40.259665 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb399a43-c060-4dc1-bf07-3c76cf919989 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:40.260045 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb399a43-c060-4dc1-bf07-3c76cf919989 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:40.260320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb399a43-c060-4dc1-bf07-3c76cf919989 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:40.283377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb399a43-c060-4dc1-bf07-3c76cf919989 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:40.290148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-cb399a43-c060-4dc1-bf07-3c76cf919989 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:40.290148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:40.290148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:40.290148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:40.290148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:40.290148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:40.290148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:40.290148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:40.290148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:40.290148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:40.290148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:40.290148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:40.290148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:40.290148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:40.290148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:40.290148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:40.291046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:40.291046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:40.291046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:40.291046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:40.291046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:40.291046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:40.291046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:40.291046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:40.291046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:40.291046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:40.291046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:40.291046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:40.291046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:40.291046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:40.291046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:40.291046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:40.291833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:40.291833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:40.291833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:40.291833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:40.291833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:40.291833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:40.291833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:40.291833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:40.291833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:40.291833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:40.291833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:40.291833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:40.297321 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 124/249] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:40 2022] GET /identity/v3/roles/member => generated 84 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:12:40.298753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34c47f1-0b01-462e-97e4-7e50f87f5a30 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:40.299934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34c47f1-0b01-462e-97e4-7e50f87f5a30 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:40.300610 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34c47f1-0b01-462e-97e4-7e50f87f5a30 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:40.310178 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 125/250] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:40 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:40.313318 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34c47f1-0b01-462e-97e4-7e50f87f5a30 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:40.409960 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669f8956-d2fb-4da1-bc51-9fdf6ee94b8a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:40.462636 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:40.479156 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e630bd-5360-4a46-8b35-7b541104b658 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:40.481814 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e630bd-5360-4a46-8b35-7b541104b658 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:40.482100 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e630bd-5360-4a46-8b35-7b541104b658 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:40.482386 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e630bd-5360-4a46-8b35-7b541104b658 None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:40.483599 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e630bd-5360-4a46-8b35-7b541104b658 None admin] RBAC: Authorizing `identity:get_project(project_id=008c6a0a4a114009bc7c692aabd0b0c8)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:40.486513 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e630bd-5360-4a46-8b35-7b541104b658 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:40.491365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 126/251] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:40 2022] GET /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8 => generated 302 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:40.493485 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ce53ab-4be0-4374-81e2-18a9999d6efa None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:40.494133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ce53ab-4be0-4374-81e2-18a9999d6efa None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:40.495793 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ce53ab-4be0-4374-81e2-18a9999d6efa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:40.515259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34c47f1-0b01-462e-97e4-7e50f87f5a30 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:40.531323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-34ce53ab-4be0-4374-81e2-18a9999d6efa None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:40.533964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-34ce53ab-4be0-4374-81e2-18a9999d6efa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:40.574100 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6630d846-0c6a-4def-aeb2-0a37a6734b7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:40.574712 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6630d846-0c6a-4def-aeb2-0a37a6734b7a None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:40.574989 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6630d846-0c6a-4def-aeb2-0a37a6734b7a None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:40.575259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6630d846-0c6a-4def-aeb2-0a37a6734b7a None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:40.576035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6630d846-0c6a-4def-aeb2-0a37a6734b7a None admin] RBAC: Adding query filter params (name=member) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:40.576337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6630d846-0c6a-4def-aeb2-0a37a6734b7a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:40.577874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6630d846-0c6a-4def-aeb2-0a37a6734b7a None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:40.591758 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 126/252] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:40 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:40.591758 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6630d846-0c6a-4def-aeb2-0a37a6734b7a None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:40.633649 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:40.643858 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 127/253] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:40.645287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ce53ab-4be0-4374-81e2-18a9999d6efa None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:40.724040 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6630d846-0c6a-4def-aeb2-0a37a6734b7a None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:40.765396 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ce53ab-4be0-4374-81e2-18a9999d6efa None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:40.794387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:40.805630 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418b4dba-9a01-4d34-90a6-23838304605a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:40.806987 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418b4dba-9a01-4d34-90a6-23838304605a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:40.807513 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418b4dba-9a01-4d34-90a6-23838304605a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:40.807979 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418b4dba-9a01-4d34-90a6-23838304605a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:40.825288 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-418b4dba-9a01-4d34-90a6-23838304605a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:40.830859 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e9bf72-c536-4097-8a2b-2d37b91b9afc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:40.831014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-418b4dba-9a01-4d34-90a6-23838304605a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:40.831014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:40.831014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:40.831014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:40.831014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:40.831014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:40.831014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:40.831014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:40.831014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:40.831014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:40.831014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:40.831014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:40.831014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:40.831014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:40.831014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:40.831014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:40.831848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:40.831848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:40.831848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:40.831848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:40.831848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:40.831848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:40.831848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:40.831848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:40.831848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:40.831848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:40.831848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:40.831848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:40.831848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:40.831848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:40.831848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:40.831848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:40.832646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:40.832646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:40.832646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:40.832646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:40.832646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:40.832646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:40.832646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:40.832646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:40.832646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:40.832646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:40.832646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:40.832646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:40.834362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e9bf72-c536-4097-8a2b-2d37b91b9afc None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:40.834362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e9bf72-c536-4097-8a2b-2d37b91b9afc None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:40.834362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e9bf72-c536-4097-8a2b-2d37b91b9afc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:40.834362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e9bf72-c536-4097-8a2b-2d37b91b9afc None admin] RBAC: Adding query filter params (role.id=902fd8d9a4ef48058e1b17e0429ede58, scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8, user.id=dba522ffd9424baf80a496f4095561b7) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:40.834362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e9bf72-c536-4097-8a2b-2d37b91b9afc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:40.834968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e9bf72-c536-4097-8a2b-2d37b91b9afc None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:40.836302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 128/254] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:12:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:40.837302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418b4dba-9a01-4d34-90a6-23838304605a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:40.848809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 127/255] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:12:40 2022] GET /identity/v3/role_assignments?user.id=dba522ffd9424baf80a496f4095561b7&scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8&role.id=902fd8d9a4ef48058e1b17e0429ede58 => generated 259 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:40.849729 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e9bf72-c536-4097-8a2b-2d37b91b9afc None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:40.960451 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418b4dba-9a01-4d34-90a6-23838304605a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:41.003532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d59ccf4-7312-4009-9e64-b5d7a34b6d55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:41.004983 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d59ccf4-7312-4009-9e64-b5d7a34b6d55 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:41.006064 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d59ccf4-7312-4009-9e64-b5d7a34b6d55 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:41.006834 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d59ccf4-7312-4009-9e64-b5d7a34b6d55 None admin] PATH_INFO: `/v3/users/0fba3b2cdd9843f0bf62fe574d3e3ec1` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:41.008304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d59ccf4-7312-4009-9e64-b5d7a34b6d55 None admin] RBAC: Authorizing `identity:get_user(user_id=0fba3b2cdd9843f0bf62fe574d3e3ec1)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:41.010685 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d59ccf4-7312-4009-9e64-b5d7a34b6d55 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:41.012737 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 129/256] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:40 2022] GET /identity/v3/users/0fba3b2cdd9843f0bf62fe574d3e3ec1 => generated 297 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:41.019653 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d59ccf4-7312-4009-9e64-b5d7a34b6d55 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:41.051155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:41.058549 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e9bf72-c536-4097-8a2b-2d37b91b9afc None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:41.087139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:41.147120 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caaddff-aaef-4cbf-bdd0-954be13bd9cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:41.148009 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caaddff-aaef-4cbf-bdd0-954be13bd9cd None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:41.148426 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caaddff-aaef-4cbf-bdd0-954be13bd9cd None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:41.148832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caaddff-aaef-4cbf-bdd0-954be13bd9cd None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:41.149667 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caaddff-aaef-4cbf-bdd0-954be13bd9cd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:41.150106 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caaddff-aaef-4cbf-bdd0-954be13bd9cd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:41.152526 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caaddff-aaef-4cbf-bdd0-954be13bd9cd None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:41.155588 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d59ccf4-7312-4009-9e64-b5d7a34b6d55 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:41.164634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 128/257] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:41.165943 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caaddff-aaef-4cbf-bdd0-954be13bd9cd None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:41.199323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13ca5a4-7e3c-4f27-89b8-0ed94cf1fc20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:41.200012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13ca5a4-7e3c-4f27-89b8-0ed94cf1fc20 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:41.200290 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13ca5a4-7e3c-4f27-89b8-0ed94cf1fc20 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:41.200555 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13ca5a4-7e3c-4f27-89b8-0ed94cf1fc20 None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:41.205776 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d13ca5a4-7e3c-4f27-89b8-0ed94cf1fc20 None admin] RBAC: Authorizing `identity:get_project(project_id=008c6a0a4a114009bc7c692aabd0b0c8)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:41.207522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d13ca5a4-7e3c-4f27-89b8-0ed94cf1fc20 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:41.210174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 130/258] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:41 2022] GET /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8 => generated 302 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:41.211013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13ca5a4-7e3c-4f27-89b8-0ed94cf1fc20 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:41.336844 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caaddff-aaef-4cbf-bdd0-954be13bd9cd None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:41.401909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebaa5a8-ae19-499d-879c-cf22b5a6aeaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:41.402914 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebaa5a8-ae19-499d-879c-cf22b5a6aeaf None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:41.404111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebaa5a8-ae19-499d-879c-cf22b5a6aeaf None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:41.405086 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebaa5a8-ae19-499d-879c-cf22b5a6aeaf None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:41.407596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13ca5a4-7e3c-4f27-89b8-0ed94cf1fc20 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:41.426468 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebaa5a8-ae19-499d-879c-cf22b5a6aeaf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:41.434701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-8ebaa5a8-ae19-499d-879c-cf22b5a6aeaf None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:12:41.434701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:41.434701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:41.434701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:41.434701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:41.434701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:41.434701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:41.434701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:41.434701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:41.434701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:41.434701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:41.434701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:41.434701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:41.434701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:41.434701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:41.434701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:41.435784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:41.435784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:41.435784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:41.435784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:41.435784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:41.435784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:41.435784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:41.435784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:41.435784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:41.435784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:41.435784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:41.435784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:41.435784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:41.435784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:41.435784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:41.435784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:41.436834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:41.436834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:41.436834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:41.436834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:41.436834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:41.436834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:41.436834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:41.436834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:41.436834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:41.436834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:41.436834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:12:41.436834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:41.440397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 129/259] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:12:41 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:41.444942 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebaa5a8-ae19-499d-879c-cf22b5a6aeaf None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:41.465392 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:41.477721 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfcf580-cf0a-4c67-b475-09379de3866f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:41.478376 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfcf580-cf0a-4c67-b475-09379de3866f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:41.478651 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfcf580-cf0a-4c67-b475-09379de3866f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:41.478919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfcf580-cf0a-4c67-b475-09379de3866f None admin] PATH_INFO: `/v3/users/30b88efcb0da48dc8df8edbeb6af20d7` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:41.485717 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfcf580-cf0a-4c67-b475-09379de3866f None admin] RBAC: Authorizing `identity:get_user(user_id=30b88efcb0da48dc8df8edbeb6af20d7)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:41.487191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfcf580-cf0a-4c67-b475-09379de3866f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:41.488266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:41.489145 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 131/260] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:41 2022] GET /identity/v3/users/30b88efcb0da48dc8df8edbeb6af20d7 => generated 283 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:41.490646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfcf580-cf0a-4c67-b475-09379de3866f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:41.617505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebaa5a8-ae19-499d-879c-cf22b5a6aeaf None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:41.667709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfcf580-cf0a-4c67-b475-09379de3866f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:41.680934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d195ed-38bd-4902-b59c-68c5e83a2c38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:41.681633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d195ed-38bd-4902-b59c-68c5e83a2c38 None admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:41.681914 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d195ed-38bd-4902-b59c-68c5e83a2c38 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:41.682187 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d195ed-38bd-4902-b59c-68c5e83a2c38 None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8/users/0fba3b2cdd9843f0bf62fe574d3e3ec1/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:41.687009 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d195ed-38bd-4902-b59c-68c5e83a2c38 None admin] RBAC: Authorizing `identity:create_grant(project_id=008c6a0a4a114009bc7c692aabd0b0c8, user_id=0fba3b2cdd9843f0bf62fe574d3e3ec1, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:41.688496 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d195ed-38bd-4902-b59c-68c5e83a2c38 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:41.712246 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 130/261] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:12:41 2022] PUT /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8/users/0fba3b2cdd9843f0bf62fe574d3e3ec1/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:12:41.712909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d195ed-38bd-4902-b59c-68c5e83a2c38 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:41.741506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08bf150-3fc6-40ee-8214-221c318cb947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:41.742142 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08bf150-3fc6-40ee-8214-221c318cb947 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:41.742419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08bf150-3fc6-40ee-8214-221c318cb947 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:41.742689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08bf150-3fc6-40ee-8214-221c318cb947 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:41.743366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08bf150-3fc6-40ee-8214-221c318cb947 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:41.743664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08bf150-3fc6-40ee-8214-221c318cb947 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:41.745066 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08bf150-3fc6-40ee-8214-221c318cb947 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:41.763666 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 132/262] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:12:41 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:41.769517 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08bf150-3fc6-40ee-8214-221c318cb947 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:41.832033 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:41.870392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d195ed-38bd-4902-b59c-68c5e83a2c38 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:41.872192 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:41.948827 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d3a0a8-6d5b-467f-a376-da620b5679a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:41.949475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d3a0a8-6d5b-467f-a376-da620b5679a0 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:41.949755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d3a0a8-6d5b-467f-a376-da620b5679a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:41.950021 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d3a0a8-6d5b-467f-a376-da620b5679a0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:41.974677 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08bf150-3fc6-40ee-8214-221c318cb947 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:42.040693 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d3a0a8-6d5b-467f-a376-da620b5679a0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:42.042472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d3a0a8-6d5b-467f-a376-da620b5679a0 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:42.069800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77a16f3-b021-424a-8061-7e76498a016f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:42.070468 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77a16f3-b021-424a-8061-7e76498a016f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:42.070756 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77a16f3-b021-424a-8061-7e76498a016f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:42.071038 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77a16f3-b021-424a-8061-7e76498a016f None admin] PATH_INFO: `/v3/users/30b88efcb0da48dc8df8edbeb6af20d7` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:42.072246 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77a16f3-b021-424a-8061-7e76498a016f None admin] RBAC: Authorizing `identity:get_user(user_id=30b88efcb0da48dc8df8edbeb6af20d7)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:42.073758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77a16f3-b021-424a-8061-7e76498a016f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:42.075670 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 133/263] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:41 2022] GET /identity/v3/users/30b88efcb0da48dc8df8edbeb6af20d7 => generated 283 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:42.080860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77a16f3-b021-424a-8061-7e76498a016f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:42.180790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:42.206070 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 131/264] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 493 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:12:42.212890 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d3a0a8-6d5b-467f-a376-da620b5679a0 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:42.319786 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77a16f3-b021-424a-8061-7e76498a016f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:42.358477 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:42.385800 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d3a0a8-6d5b-467f-a376-da620b5679a0 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:42.401772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed2af71-cf94-400a-970a-4b3f657bc823 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:42.402732 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed2af71-cf94-400a-970a-4b3f657bc823 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:42.403208 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed2af71-cf94-400a-970a-4b3f657bc823 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:42.403673 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed2af71-cf94-400a-970a-4b3f657bc823 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:42.462575 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d08e66a-4971-4369-a51a-ed1d74a5410c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:42.463227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d08e66a-4971-4369-a51a-ed1d74a5410c None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:42.463505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d08e66a-4971-4369-a51a-ed1d74a5410c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:42.463774 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d08e66a-4971-4369-a51a-ed1d74a5410c None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:42.469765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d08e66a-4971-4369-a51a-ed1d74a5410c None admin] RBAC: Authorizing `identity:get_project(project_id=008c6a0a4a114009bc7c692aabd0b0c8)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:42.471601 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d08e66a-4971-4369-a51a-ed1d74a5410c None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:42.476085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 132/265] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:42 2022] GET /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8 => generated 302 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:42.478382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c90e928-c2b1-4018-9da6-a032b3f3f1ee None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:42.483037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c90e928-c2b1-4018-9da6-a032b3f3f1ee None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:42.483533 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c90e928-c2b1-4018-9da6-a032b3f3f1ee None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:42.491678 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 133/266] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:42 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:42.492776 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c90e928-c2b1-4018-9da6-a032b3f3f1ee None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:42.535560 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed2af71-cf94-400a-970a-4b3f657bc823 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:42.538017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed2af71-cf94-400a-970a-4b3f657bc823 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:42.563640 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:42.634175 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c90e928-c2b1-4018-9da6-a032b3f3f1ee None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:42.664838 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 134/267] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 580 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:12:42.664838 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4b4579-e8ef-48b3-ba51-5d580aab075f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:42.664838 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4b4579-e8ef-48b3-ba51-5d580aab075f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:42.664838 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4b4579-e8ef-48b3-ba51-5d580aab075f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:42.681326 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-1a4b4579-e8ef-48b3-ba51-5d580aab075f None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:42.682637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1a4b4579-e8ef-48b3-ba51-5d580aab075f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:42.686434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb9d52f-6fa0-49d9-9d10-509c04328bbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:42.687217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb9d52f-6fa0-49d9-9d10-509c04328bbc None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:42.687639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb9d52f-6fa0-49d9-9d10-509c04328bbc None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:42.688049 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb9d52f-6fa0-49d9-9d10-509c04328bbc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:42.688916 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb9d52f-6fa0-49d9-9d10-509c04328bbc None admin] RBAC: Adding query filter params (role.id=d17954e7ce544f0d90a512340232c60e, scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8, user.id=30b88efcb0da48dc8df8edbeb6af20d7) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:42.689403 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb9d52f-6fa0-49d9-9d10-509c04328bbc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:42.691511 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb9d52f-6fa0-49d9-9d10-509c04328bbc None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:42.697773 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:42.701078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 134/268] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:12:42 2022] GET /identity/v3/role_assignments?user.id=30b88efcb0da48dc8df8edbeb6af20d7&scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8&role.id=d17954e7ce544f0d90a512340232c60e => generated 608 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:42.702545 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3a6289-08f7-4c20-9231-c18abe57528c None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:42.703028 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3a6289-08f7-4c20-9231-c18abe57528c None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:42.703462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3a6289-08f7-4c20-9231-c18abe57528c None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:42.704879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 135/269] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:42.705787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3a6289-08f7-4c20-9231-c18abe57528c None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:42.832974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3a6289-08f7-4c20-9231-c18abe57528c None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:42.851555 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:42.853812 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 135/270] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:42.855101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53956e59-e079-4441-b229-9e2df835a5e3 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:42.860880 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53956e59-e079-4441-b229-9e2df835a5e3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:42.860880 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53956e59-e079-4441-b229-9e2df835a5e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:42.898585 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35285c9-4cc9-4fc0-837a-c4a10cd7c203 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:42.899589 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35285c9-4cc9-4fc0-837a-c4a10cd7c203 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:42.900051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35285c9-4cc9-4fc0-837a-c4a10cd7c203 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:42.900504 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35285c9-4cc9-4fc0-837a-c4a10cd7c203 None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:42.901757 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-53956e59-e079-4441-b229-9e2df835a5e3 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:42.903125 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35285c9-4cc9-4fc0-837a-c4a10cd7c203 None admin] RBAC: Authorizing `identity:get_project(project_id=008c6a0a4a114009bc7c692aabd0b0c8)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:42.904695 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-53956e59-e079-4441-b229-9e2df835a5e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:42.906411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35285c9-4cc9-4fc0-837a-c4a10cd7c203 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:42.910666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 136/271] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:42 2022] GET /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8 => generated 302 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:42.931652 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35285c9-4cc9-4fc0-837a-c4a10cd7c203 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:43.046400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 136/272] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:43.050926 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35285c9-4cc9-4fc0-837a-c4a10cd7c203 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:43.086745 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402b37f9-5b5f-447a-8391-1015d11b3fda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:43.087626 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402b37f9-5b5f-447a-8391-1015d11b3fda None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:43.088088 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402b37f9-5b5f-447a-8391-1015d11b3fda None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:43.088553 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402b37f9-5b5f-447a-8391-1015d11b3fda None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:43.089481 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402b37f9-5b5f-447a-8391-1015d11b3fda None admin] RBAC: Adding query filter params (role.id=a9e862ac528245ca8363bccc5bf35287, scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8, user.id=30b88efcb0da48dc8df8edbeb6af20d7) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:43.089965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402b37f9-5b5f-447a-8391-1015d11b3fda None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:43.091562 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402b37f9-5b5f-447a-8391-1015d11b3fda None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:43.098024 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 137/273] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:12:42 2022] GET /identity/v3/role_assignments?user.id=30b88efcb0da48dc8df8edbeb6af20d7&scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8&role.id=a9e862ac528245ca8363bccc5bf35287 => generated 608 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:43.207134 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53956e59-e079-4441-b229-9e2df835a5e3 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:43.220629 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec40e684-11ed-4c5e-a685-8e4be7a67652 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:43.221377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec40e684-11ed-4c5e-a685-8e4be7a67652 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:43.221377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec40e684-11ed-4c5e-a685-8e4be7a67652 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:43.222599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 138/274] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:12:43.229737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039f4de4-cd9a-46dd-a8ec-5598ff845d5e None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:43.230354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039f4de4-cd9a-46dd-a8ec-5598ff845d5e None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:43.230825 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039f4de4-cd9a-46dd-a8ec-5598ff845d5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:43.270146 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-039f4de4-cd9a-46dd-a8ec-5598ff845d5e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:43.271821 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-039f4de4-cd9a-46dd-a8ec-5598ff845d5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:43.329469 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:43.341392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:43.367002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53956e59-e079-4441-b229-9e2df835a5e3 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:43.397759 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 139/275] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:43.404667 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a964e861-95a7-4d03-857f-1609fc1ae2f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:43.405341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a964e861-95a7-4d03-857f-1609fc1ae2f7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:43.405622 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a964e861-95a7-4d03-857f-1609fc1ae2f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:43.405895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a964e861-95a7-4d03-857f-1609fc1ae2f7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:43.407002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a964e861-95a7-4d03-857f-1609fc1ae2f7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:43.408451 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a964e861-95a7-4d03-857f-1609fc1ae2f7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:43.410234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 137/276] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:12:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:43.415961 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039f4de4-cd9a-46dd-a8ec-5598ff845d5e None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:43.425621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a964e861-95a7-4d03-857f-1609fc1ae2f7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:43.539573 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039f4de4-cd9a-46dd-a8ec-5598ff845d5e None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:43.554644 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a964e861-95a7-4d03-857f-1609fc1ae2f7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:43.589727 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f4b5ea-8c37-4acb-8e45-73c5a3e9ecee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:43.590327 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f4b5ea-8c37-4acb-8e45-73c5a3e9ecee None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:43.590698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f4b5ea-8c37-4acb-8e45-73c5a3e9ecee None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:43.590972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f4b5ea-8c37-4acb-8e45-73c5a3e9ecee None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:43.591762 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f4b5ea-8c37-4acb-8e45-73c5a3e9ecee None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:43.593182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f4b5ea-8c37-4acb-8e45-73c5a3e9ecee None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:43.598995 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7246e6-16e6-495b-b173-6cea1a058d40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:43.599713 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7246e6-16e6-495b-b173-6cea1a058d40 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:43.599993 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7246e6-16e6-495b-b173-6cea1a058d40 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:43.600264 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7246e6-16e6-495b-b173-6cea1a058d40 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:43.616270 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7246e6-16e6-495b-b173-6cea1a058d40 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:43.618697 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 140/277] 10.222.0.85 () {60 vars in 1132 bytes} [Tue Aug 30 04:12:43 2022] POST /identity/v3/users => generated 293 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:12:43.621734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-dd7246e6-16e6-495b-b173-6cea1a058d40 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:43.621734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:43.621734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:43.621734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:43.621734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:43.621734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:43.621734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:43.621734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:43.621734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:43.621734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:43.621734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:43.621734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:43.621734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:43.621734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:43.621734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:43.621734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:43.622616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:43.622616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:43.622616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:43.622616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:43.622616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:43.622616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:43.622616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:43.622616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:43.622616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:43.622616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:43.622616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:43.622616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:43.622616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:43.622616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:43.622616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:43.622616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:43.623401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:43.623401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:43.623401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:43.623401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:43.623401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:43.623401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:43.623401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:43.623401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:43.623401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:43.623401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:43.623401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:43.623401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:43.628229 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 138/278] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:43 2022] GET /identity/v3/roles/member => generated 84 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:12:43.629209 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f4b5ea-8c37-4acb-8e45-73c5a3e9ecee None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:43.679708 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:43.693186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7246e6-16e6-495b-b173-6cea1a058d40 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:43.703005 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:43.770018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f4b5ea-8c37-4acb-8e45-73c5a3e9ecee None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:43.814345 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52605d41-52fd-4a96-9804-bfbe3b529302 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:43.814971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52605d41-52fd-4a96-9804-bfbe3b529302 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:43.815255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52605d41-52fd-4a96-9804-bfbe3b529302 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:43.815528 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52605d41-52fd-4a96-9804-bfbe3b529302 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:43.816543 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52605d41-52fd-4a96-9804-bfbe3b529302 None admin] RBAC: Adding query filter params (name=member) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:43.816732 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52605d41-52fd-4a96-9804-bfbe3b529302 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:43.818768 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52605d41-52fd-4a96-9804-bfbe3b529302 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:43.822173 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7246e6-16e6-495b-b173-6cea1a058d40 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:43.832695 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 141/279] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:43 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:43.840563 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52605d41-52fd-4a96-9804-bfbe3b529302 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:43.867636 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dffc219-9ac0-49ec-b07a-5dfb1f22ebf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:43.868223 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dffc219-9ac0-49ec-b07a-5dfb1f22ebf3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:43.869171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dffc219-9ac0-49ec-b07a-5dfb1f22ebf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:43.869171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dffc219-9ac0-49ec-b07a-5dfb1f22ebf3 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:43.883615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dffc219-9ac0-49ec-b07a-5dfb1f22ebf3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:43.888639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-3dffc219-9ac0-49ec-b07a-5dfb1f22ebf3 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:12:43.888639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:43.888639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:43.888639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:43.888639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:43.888639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:43.888639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:43.888639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:43.888639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:43.888639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:43.888639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:43.888639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:43.888639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:43.888639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:43.888639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:43.888639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:43.889549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:43.889549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:43.889549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:43.889549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:43.889549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:43.889549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:43.889549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:43.889549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:43.889549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:43.889549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:43.889549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:43.889549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:43.889549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:43.889549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:43.889549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:43.889549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:43.890368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:43.890368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:43.890368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:43.890368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:43.890368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:43.890368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:43.890368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:43.890368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:43.890368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:43.890368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:43.890368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:12:43.890368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:43.891313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 139/280] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:43 2022] GET /identity/v3/roles/reader => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:12:43.898872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dffc219-9ac0-49ec-b07a-5dfb1f22ebf3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:43.973691 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:43.982195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:43.986484 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52605d41-52fd-4a96-9804-bfbe3b529302 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:44.023473 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dffc219-9ac0-49ec-b07a-5dfb1f22ebf3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:44.035527 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ee98f1-6456-467b-87cb-958645606ee8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:44.036655 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ee98f1-6456-467b-87cb-958645606ee8 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:44.037802 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ee98f1-6456-467b-87cb-958645606ee8 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:44.038232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ee98f1-6456-467b-87cb-958645606ee8 None admin] PATH_INFO: `/v3/users/db8f383f5f184597955d112061db403c` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:44.057060 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ee98f1-6456-467b-87cb-958645606ee8 None admin] RBAC: Authorizing `identity:get_user(user_id=db8f383f5f184597955d112061db403c)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:44.059447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ee98f1-6456-467b-87cb-958645606ee8 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:44.061310 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 142/281] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:43 2022] GET /identity/v3/users/db8f383f5f184597955d112061db403c => generated 293 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:44.062986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fba36f3-fbd5-446f-bcbd-af8d674b19dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:44.063601 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fba36f3-fbd5-446f-bcbd-af8d674b19dd None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:44.063876 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fba36f3-fbd5-446f-bcbd-af8d674b19dd None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:44.064146 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fba36f3-fbd5-446f-bcbd-af8d674b19dd None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:44.064810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fba36f3-fbd5-446f-bcbd-af8d674b19dd None admin] RBAC: Adding query filter params (name=reader) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:44.065750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fba36f3-fbd5-446f-bcbd-af8d674b19dd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:44.066709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fba36f3-fbd5-446f-bcbd-af8d674b19dd None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:44.067564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ee98f1-6456-467b-87cb-958645606ee8 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:44.077715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 140/282] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:43 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:44.082684 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fba36f3-fbd5-446f-bcbd-af8d674b19dd None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:44.212732 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ee98f1-6456-467b-87cb-958645606ee8 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:44.221370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fba36f3-fbd5-446f-bcbd-af8d674b19dd None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:44.269127 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:44.275653 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e424733-42c6-4f46-8cad-84d7a78483be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:44.276272 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e424733-42c6-4f46-8cad-84d7a78483be None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:44.276546 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e424733-42c6-4f46-8cad-84d7a78483be None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:44.276812 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e424733-42c6-4f46-8cad-84d7a78483be None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:44.277564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e424733-42c6-4f46-8cad-84d7a78483be None admin] RBAC: Adding query filter params (role.id=ae0cef8120cf4dfc899c30803243a851, user.id=db8f383f5f184597955d112061db403c, scope.system=all) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:44.277869 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e424733-42c6-4f46-8cad-84d7a78483be None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:44.279311 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e424733-42c6-4f46-8cad-84d7a78483be None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:44.286394 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c794fa66-00bd-4b79-bb78-05b24b728f23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:44.287591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c794fa66-00bd-4b79-bb78-05b24b728f23 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:44.287873 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c794fa66-00bd-4b79-bb78-05b24b728f23 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:44.288252 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c794fa66-00bd-4b79-bb78-05b24b728f23 None admin] PATH_INFO: `/v3/users/dba522ffd9424baf80a496f4095561b7` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:44.290883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c794fa66-00bd-4b79-bb78-05b24b728f23 None admin] RBAC: Authorizing `identity:get_user(user_id=dba522ffd9424baf80a496f4095561b7)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:44.292347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c794fa66-00bd-4b79-bb78-05b24b728f23 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:44.294120 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 141/283] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:44 2022] GET /identity/v3/users/dba522ffd9424baf80a496f4095561b7 => generated 297 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:44.295293 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adfdb73-831b-48e8-a697-79725bfc5e37 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:44.295511 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 143/284] 10.222.0.85 () {56 vars in 1308 bytes} [Tue Aug 30 04:12:44 2022] GET /identity/v3/role_assignments?user.id=db8f383f5f184597955d112061db403c&scope.system=all&role.id=ae0cef8120cf4dfc899c30803243a851 => generated 226 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:44.295612 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adfdb73-831b-48e8-a697-79725bfc5e37 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:44.297192 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adfdb73-831b-48e8-a697-79725bfc5e37 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:44.297303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 142/285] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:44.301734 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e424733-42c6-4f46-8cad-84d7a78483be None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:44.307783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98838dfd-b4a0-477a-b2d3-fbd90500d043 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:44.308098 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98838dfd-b4a0-477a-b2d3-fbd90500d043 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:44.308339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98838dfd-b4a0-477a-b2d3-fbd90500d043 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:44.337053 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:44.337647 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-98838dfd-b4a0-477a-b2d3-fbd90500d043 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:44.339201 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-98838dfd-b4a0-477a-b2d3-fbd90500d043 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:44.454704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e424733-42c6-4f46-8cad-84d7a78483be None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:44.493930 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7017e2f-3524-4f40-8a17-c0e4164bb25b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:44.494917 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7017e2f-3524-4f40-8a17-c0e4164bb25b None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:44.495419 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7017e2f-3524-4f40-8a17-c0e4164bb25b None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:44.495895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7017e2f-3524-4f40-8a17-c0e4164bb25b None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:44.497254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7017e2f-3524-4f40-8a17-c0e4164bb25b None admin] RBAC: Authorizing `identity:get_project(project_id=008c6a0a4a114009bc7c692aabd0b0c8)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:44.499292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7017e2f-3524-4f40-8a17-c0e4164bb25b None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:44.501576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 144/286] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:44 2022] GET /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8 => generated 302 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:44.507668 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7017e2f-3524-4f40-8a17-c0e4164bb25b None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:44.550355 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 143/287] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 248 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:44.642462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:44.661030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7017e2f-3524-4f40-8a17-c0e4164bb25b None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:44.708919 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aad3aef-3cb5-425e-8608-c3d031ac0b32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:44.709597 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aad3aef-3cb5-425e-8608-c3d031ac0b32 None admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:44.709866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aad3aef-3cb5-425e-8608-c3d031ac0b32 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:44.710134 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aad3aef-3cb5-425e-8608-c3d031ac0b32 None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8/users/dba522ffd9424baf80a496f4095561b7/roles/902fd8d9a4ef48058e1b17e0429ede58` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:44.712111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aad3aef-3cb5-425e-8608-c3d031ac0b32 None admin] RBAC: Authorizing `identity:create_grant(project_id=008c6a0a4a114009bc7c692aabd0b0c8, user_id=dba522ffd9424baf80a496f4095561b7, role_id=902fd8d9a4ef48058e1b17e0429ede58)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:44.713578 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aad3aef-3cb5-425e-8608-c3d031ac0b32 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:44.733228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 145/288] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:12:44 2022] PUT /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8/users/dba522ffd9424baf80a496f4095561b7/roles/902fd8d9a4ef48058e1b17e0429ede58 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:12:44.876822 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98838dfd-b4a0-477a-b2d3-fbd90500d043 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:45.000373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98838dfd-b4a0-477a-b2d3-fbd90500d043 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:45.028418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:45.046693 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5560eda-4552-47c3-a7b8-37ee4c3cfeb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:45.047342 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5560eda-4552-47c3-a7b8-37ee4c3cfeb3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:45.047617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5560eda-4552-47c3-a7b8-37ee4c3cfeb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:45.047885 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5560eda-4552-47c3-a7b8-37ee4c3cfeb3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:45.060420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5560eda-4552-47c3-a7b8-37ee4c3cfeb3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:45.065347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-d5560eda-4552-47c3-a7b8-37ee4c3cfeb3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:45.065347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:45.065347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:45.065347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:45.065347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:45.065347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:45.065347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:45.065347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:45.065347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:45.065347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:45.065347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:45.065347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:45.065347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:45.065347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:45.065347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:45.065347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:45.066228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:45.066228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:45.066228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:45.066228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:45.066228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:45.066228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:45.066228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:45.066228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:45.066228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:45.066228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:45.066228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:45.066228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:45.066228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:45.066228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:45.066228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:45.066228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:45.067102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:45.067102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:45.067102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:45.067102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:45.067102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:45.067102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:45.067102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:45.067102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:45.067102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:45.067102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:45.067102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:45.067102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:45.067102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 144/289] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:44 2022] GET /identity/v3/roles/member => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:45.072536 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aad3aef-3cb5-425e-8608-c3d031ac0b32 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:45.187289 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aad3aef-3cb5-425e-8608-c3d031ac0b32 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:45.236128 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1e986f-8061-4f2c-a32b-64896317faba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:45.236736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1e986f-8061-4f2c-a32b-64896317faba None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:45.237013 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1e986f-8061-4f2c-a32b-64896317faba None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:45.237304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1e986f-8061-4f2c-a32b-64896317faba None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:45.238124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1e986f-8061-4f2c-a32b-64896317faba None admin] RBAC: Adding query filter params (name=member) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:45.238426 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1e986f-8061-4f2c-a32b-64896317faba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:45.239794 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1e986f-8061-4f2c-a32b-64896317faba None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:45.252948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 146/290] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:45.254512 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5560eda-4552-47c3-a7b8-37ee4c3cfeb3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:45.375366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5560eda-4552-47c3-a7b8-37ee4c3cfeb3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:45.411230 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44b297c-2a57-4a7d-b786-6e4c767a0a0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:45.412393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44b297c-2a57-4a7d-b786-6e4c767a0a0d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:45.412669 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44b297c-2a57-4a7d-b786-6e4c767a0a0d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:45.412935 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44b297c-2a57-4a7d-b786-6e4c767a0a0d None admin] PATH_INFO: `/v3/users/0fba3b2cdd9843f0bf62fe574d3e3ec1` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:45.414109 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44b297c-2a57-4a7d-b786-6e4c767a0a0d None admin] RBAC: Authorizing `identity:get_user(user_id=0fba3b2cdd9843f0bf62fe574d3e3ec1)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:45.415614 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44b297c-2a57-4a7d-b786-6e4c767a0a0d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:45.417335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 145/291] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:45 2022] GET /identity/v3/users/0fba3b2cdd9843f0bf62fe574d3e3ec1 => generated 297 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:45.423373 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1e986f-8061-4f2c-a32b-64896317faba None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:45.433739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:45.543217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1e986f-8061-4f2c-a32b-64896317faba None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:45.578708 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a76dba-f264-4a8b-a09c-7062e007b25e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:45.579326 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a76dba-f264-4a8b-a09c-7062e007b25e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:45.579609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a76dba-f264-4a8b-a09c-7062e007b25e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:45.579878 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a76dba-f264-4a8b-a09c-7062e007b25e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:45.669857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a76dba-f264-4a8b-a09c-7062e007b25e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:45.671330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a76dba-f264-4a8b-a09c-7062e007b25e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:45.693894 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:45.785872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 147/292] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:12:45.792293 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44b297c-2a57-4a7d-b786-6e4c767a0a0d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:45.870345 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:45.935871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44b297c-2a57-4a7d-b786-6e4c767a0a0d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:45.993764 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b96c769-d5a7-440c-8e41-d77aa08792a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:45.994809 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b96c769-d5a7-440c-8e41-d77aa08792a5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:45.995307 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b96c769-d5a7-440c-8e41-d77aa08792a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:45.995781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b96c769-d5a7-440c-8e41-d77aa08792a5 None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:45.997252 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b96c769-d5a7-440c-8e41-d77aa08792a5 None admin] RBAC: Authorizing `identity:get_project(project_id=008c6a0a4a114009bc7c692aabd0b0c8)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:45.999055 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b96c769-d5a7-440c-8e41-d77aa08792a5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:46.001284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 146/293] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:45 2022] GET /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8 => generated 302 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:46.007774 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a76dba-f264-4a8b-a09c-7062e007b25e None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:46.115854 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a76dba-f264-4a8b-a09c-7062e007b25e None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:46.127283 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:46.168502 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4da55fa-9308-49b3-a83c-ce4d4f2f1543 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:46.169151 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4da55fa-9308-49b3-a83c-ce4d4f2f1543 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:46.169438 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4da55fa-9308-49b3-a83c-ce4d4f2f1543 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:46.169716 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4da55fa-9308-49b3-a83c-ce4d4f2f1543 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:46.170456 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4da55fa-9308-49b3-a83c-ce4d4f2f1543 None admin] RBAC: Adding query filter params (role.id=ae0cef8120cf4dfc899c30803243a851, scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8, user.id=0fba3b2cdd9843f0bf62fe574d3e3ec1) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:46.170767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4da55fa-9308-49b3-a83c-ce4d4f2f1543 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:46.172218 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4da55fa-9308-49b3-a83c-ce4d4f2f1543 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:46.182065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 148/294] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:12:46 2022] GET /identity/v3/role_assignments?user.id=0fba3b2cdd9843f0bf62fe574d3e3ec1&scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8&role.id=ae0cef8120cf4dfc899c30803243a851 => generated 608 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:46.442646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0907d66b-dcda-4b77-a6b6-4bf54c66467d None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:46.443123 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0907d66b-dcda-4b77-a6b6-4bf54c66467d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:46.443350 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0907d66b-dcda-4b77-a6b6-4bf54c66467d None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:46.444797 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 147/295] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:46.450363 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0adc0a-eb40-4ef5-8258-cef920754c91 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:46.450654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0adc0a-eb40-4ef5-8258-cef920754c91 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:46.450934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0adc0a-eb40-4ef5-8258-cef920754c91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:46.454879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8218d7c-e607-4dfb-8f62-74da8aecf67d None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:46.455163 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8218d7c-e607-4dfb-8f62-74da8aecf67d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:46.455441 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8218d7c-e607-4dfb-8f62-74da8aecf67d None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:46.456912 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 148/296] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:46.463894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca792b08-5a95-4de2-a23e-a5404d392626 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:46.464180 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca792b08-5a95-4de2-a23e-a5404d392626 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:46.464454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca792b08-5a95-4de2-a23e-a5404d392626 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:46.483275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-dc0adc0a-eb40-4ef5-8258-cef920754c91 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:46.484542 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-dc0adc0a-eb40-4ef5-8258-cef920754c91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:46.485882 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ca792b08-5a95-4de2-a23e-a5404d392626 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:46.487941 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ca792b08-5a95-4de2-a23e-a5404d392626 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:46.570116 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 149/297] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:46.570401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 149/298] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:46.621779 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b8e25c-0421-484e-b1fd-049213801e29 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:46.622080 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b8e25c-0421-484e-b1fd-049213801e29 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:46.622343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b8e25c-0421-484e-b1fd-049213801e29 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:46.623829 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 150/299] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:46.629259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e07594-9413-4aa4-947c-d779bd7ff66e None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:46.629547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e07594-9413-4aa4-947c-d779bd7ff66e None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:46.629822 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e07594-9413-4aa4-947c-d779bd7ff66e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:46.651462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-95e07594-9413-4aa4-947c-d779bd7ff66e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:46.652648 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-95e07594-9413-4aa4-947c-d779bd7ff66e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:46.727885 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 150/300] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:46.856202 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b8e25c-0421-484e-b1fd-049213801e29 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:46.859926 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e07594-9413-4aa4-947c-d779bd7ff66e None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:46.898503 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:46.918130 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:46.971698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b8e25c-0421-484e-b1fd-049213801e29 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:46.984548 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e07594-9413-4aa4-947c-d779bd7ff66e None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:47.010470 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca4788e-6381-464f-bf30-ab98f578e66b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:47.011106 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca4788e-6381-464f-bf30-ab98f578e66b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:47.011379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca4788e-6381-464f-bf30-ab98f578e66b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:47.011650 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca4788e-6381-464f-bf30-ab98f578e66b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:47.023555 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca4788e-6381-464f-bf30-ab98f578e66b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:47.024490 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f043e20-eb1e-4b85-9ced-561b67782197 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:47.025150 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f043e20-eb1e-4b85-9ced-561b67782197 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:47.025425 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f043e20-eb1e-4b85-9ced-561b67782197 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:47.025695 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f043e20-eb1e-4b85-9ced-561b67782197 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:47.026787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f043e20-eb1e-4b85-9ced-561b67782197 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:47.028219 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f043e20-eb1e-4b85-9ced-561b67782197 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:47.028415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-0ca4788e-6381-464f-bf30-ab98f578e66b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:12:47.028415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:47.028415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:47.028415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:47.028415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:47.028415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:47.028415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:47.028415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:47.028415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:47.028415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:47.028415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:47.028415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:47.028415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:47.028415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:47.028415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:47.028415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:47.029278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:47.029278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:47.029278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:47.029278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:47.029278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:47.029278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:47.029278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:47.029278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:47.029278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:47.029278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:47.029278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:47.029278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:47.029278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:47.029278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:47.029278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:47.029278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:47.030059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:47.030059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:47.030059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:47.030059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:47.030059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:47.030059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:47.030059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:47.030059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:47.030059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:47.030059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:47.030059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:12:47.030059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:47.030686 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 151/301] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:12:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:47.031298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f043e20-eb1e-4b85-9ced-561b67782197 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:47.037217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 151/302] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:46 2022] GET /identity/v3/roles/reader => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:47.037217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca4788e-6381-464f-bf30-ab98f578e66b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:47.158160 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca4788e-6381-464f-bf30-ab98f578e66b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:47.160027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:47.164601 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f043e20-eb1e-4b85-9ced-561b67782197 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:47.174545 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:47.197551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ddb4d1-bfbd-4aab-84f4-99a6168790d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:47.198159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ddb4d1-bfbd-4aab-84f4-99a6168790d6 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:47.198435 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ddb4d1-bfbd-4aab-84f4-99a6168790d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:47.198703 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ddb4d1-bfbd-4aab-84f4-99a6168790d6 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:47.199364 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ddb4d1-bfbd-4aab-84f4-99a6168790d6 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:47.199662 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ddb4d1-bfbd-4aab-84f4-99a6168790d6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:47.200053 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec51c089-4301-4fa8-b986-62bc0e82be8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:47.200682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec51c089-4301-4fa8-b986-62bc0e82be8c None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:47.200965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec51c089-4301-4fa8-b986-62bc0e82be8c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:47.201123 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ddb4d1-bfbd-4aab-84f4-99a6168790d6 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:47.201261 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec51c089-4301-4fa8-b986-62bc0e82be8c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:47.209305 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 152/303] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:47 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:47.209869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ddb4d1-bfbd-4aab-84f4-99a6168790d6 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:47.211404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec51c089-4301-4fa8-b986-62bc0e82be8c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:47.216298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-ec51c089-4301-4fa8-b986-62bc0e82be8c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:47.216298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:47.216298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:47.216298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:47.216298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:47.216298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:47.216298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:47.216298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:47.216298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:47.216298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:47.216298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:47.216298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:47.216298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:47.216298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:47.216298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:47.216298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:47.217257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:47.217257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:47.217257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:47.217257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:47.217257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:47.217257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:47.217257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:47.217257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:47.217257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:47.217257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:47.217257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:47.217257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:47.217257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:47.217257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:47.217257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:47.217257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:47.218149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:47.218149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:47.218149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:47.218149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:47.218149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:47.218149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:47.218149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:47.218149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:47.218149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:47.218149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:47.218149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:47.218149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:47.218149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 152/304] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:47.219473 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc8c9ea-7757-4655-8558-f7e610d6cd35 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:47.219473 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc8c9ea-7757-4655-8558-f7e610d6cd35 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:47.219473 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc8c9ea-7757-4655-8558-f7e610d6cd35 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:47.225731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 153/305] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:47.225731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc8c9ea-7757-4655-8558-f7e610d6cd35 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:47.313856 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ddb4d1-bfbd-4aab-84f4-99a6168790d6 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:47.325244 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc8c9ea-7757-4655-8558-f7e610d6cd35 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:47.347013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff9eb9d-0268-4e4a-b603-30fc1d29a610 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:47.347618 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff9eb9d-0268-4e4a-b603-30fc1d29a610 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:47.347890 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff9eb9d-0268-4e4a-b603-30fc1d29a610 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:47.348160 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff9eb9d-0268-4e4a-b603-30fc1d29a610 None admin] PATH_INFO: `/v3/groups` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:47.348775 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff9eb9d-0268-4e4a-b603-30fc1d29a610 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:47.350258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff9eb9d-0268-4e4a-b603-30fc1d29a610 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:47.358586 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b272a3f-13bf-4596-83af-a8cffca46ce9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:47.359410 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b272a3f-13bf-4596-83af-a8cffca46ce9 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:47.359690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b272a3f-13bf-4596-83af-a8cffca46ce9 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:47.359981 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b272a3f-13bf-4596-83af-a8cffca46ce9 None admin] PATH_INFO: `/v3/users/e8d54ce3bc0746c2a48bbda5024200b7` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:47.366511 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 153/306] 10.222.0.85 () {60 vars in 1134 bytes} [Tue Aug 30 04:12:47 2022] POST /identity/v3/groups => generated 229 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:12:47.367412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff9eb9d-0268-4e4a-b603-30fc1d29a610 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:47.377733 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b272a3f-13bf-4596-83af-a8cffca46ce9 None admin] RBAC: Authorizing `identity:get_user(user_id=e8d54ce3bc0746c2a48bbda5024200b7)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:47.379245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b272a3f-13bf-4596-83af-a8cffca46ce9 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:47.382086 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 154/307] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:47 2022] GET /identity/v3/users/e8d54ce3bc0746c2a48bbda5024200b7 => generated 293 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:47.383141 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca0c5b9-eb6e-4943-9115-9a422e6c9171 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:47.383426 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca0c5b9-eb6e-4943-9115-9a422e6c9171 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:47.383698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca0c5b9-eb6e-4943-9115-9a422e6c9171 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:47.402938 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-7ca0c5b9-eb6e-4943-9115-9a422e6c9171 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:47.405118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-7ca0c5b9-eb6e-4943-9115-9a422e6c9171 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:47.419105 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:47.423920 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:47.496485 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 155/308] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:47.497381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca0c5b9-eb6e-4943-9115-9a422e6c9171 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:47.497942 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff9eb9d-0268-4e4a-b603-30fc1d29a610 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:47.540578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baade6cd-2ddc-497c-b2ab-c2e7078aa659 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:47.541982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baade6cd-2ddc-497c-b2ab-c2e7078aa659 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:47.541982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baade6cd-2ddc-497c-b2ab-c2e7078aa659 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:47.541982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baade6cd-2ddc-497c-b2ab-c2e7078aa659 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:47.543859 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baade6cd-2ddc-497c-b2ab-c2e7078aa659 None admin] RBAC: Adding query filter params (name=member) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:47.543859 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baade6cd-2ddc-497c-b2ab-c2e7078aa659 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:47.546029 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baade6cd-2ddc-497c-b2ab-c2e7078aa659 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:47.556248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 154/309] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:47.563618 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baade6cd-2ddc-497c-b2ab-c2e7078aa659 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:47.630207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca0c5b9-eb6e-4943-9115-9a422e6c9171 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:47.670417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89152eef-181e-47a8-9a18-0bf149734492 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:47.671081 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89152eef-181e-47a8-9a18-0bf149734492 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:47.671367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89152eef-181e-47a8-9a18-0bf149734492 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:47.671639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89152eef-181e-47a8-9a18-0bf149734492 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:47.672432 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89152eef-181e-47a8-9a18-0bf149734492 None admin] RBAC: Adding query filter params (role.id=902fd8d9a4ef48058e1b17e0429ede58, user.id=e8d54ce3bc0746c2a48bbda5024200b7, scope.system=all) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:47.672745 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89152eef-181e-47a8-9a18-0bf149734492 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:47.674355 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89152eef-181e-47a8-9a18-0bf149734492 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:47.680414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 156/310] 10.222.0.85 () {56 vars in 1308 bytes} [Tue Aug 30 04:12:47 2022] GET /identity/v3/role_assignments?user.id=e8d54ce3bc0746c2a48bbda5024200b7&scope.system=all&role.id=902fd8d9a4ef48058e1b17e0429ede58 => generated 226 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:47.692164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:47.700484 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baade6cd-2ddc-497c-b2ab-c2e7078aa659 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:47.739104 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7728ec18-1cb3-411d-9396-0264963c6539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:47.740039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7728ec18-1cb3-411d-9396-0264963c6539 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:47.740459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7728ec18-1cb3-411d-9396-0264963c6539 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:47.740872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7728ec18-1cb3-411d-9396-0264963c6539 None admin] PATH_INFO: `/v3/users/db8f383f5f184597955d112061db403c` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:47.742360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7728ec18-1cb3-411d-9396-0264963c6539 None admin] RBAC: Authorizing `identity:get_user(user_id=db8f383f5f184597955d112061db403c)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:47.743986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7728ec18-1cb3-411d-9396-0264963c6539 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:47.746157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 155/311] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:47 2022] GET /identity/v3/users/db8f383f5f184597955d112061db403c => generated 293 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:47.754363 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89152eef-181e-47a8-9a18-0bf149734492 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:47.775134 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:47.790989 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7728ec18-1cb3-411d-9396-0264963c6539 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:47.889806 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89152eef-181e-47a8-9a18-0bf149734492 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:47.912357 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7728ec18-1cb3-411d-9396-0264963c6539 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:47.936625 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1be1f6-6257-431e-b6a0-b807c7221c48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:47.937309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1be1f6-6257-431e-b6a0-b807c7221c48 None admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:47.937590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1be1f6-6257-431e-b6a0-b807c7221c48 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:47.937863 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1be1f6-6257-431e-b6a0-b807c7221c48 None admin] PATH_INFO: `/v3/system/users/db8f383f5f184597955d112061db403c/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:47.939819 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1be1f6-6257-431e-b6a0-b807c7221c48 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=db8f383f5f184597955d112061db403c, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:47.941698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1be1f6-6257-431e-b6a0-b807c7221c48 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:47.955037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d240e13-a824-4d40-8f11-37f68087d62d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:47.955680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d240e13-a824-4d40-8f11-37f68087d62d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:47.956012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d240e13-a824-4d40-8f11-37f68087d62d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:47.956287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d240e13-a824-4d40-8f11-37f68087d62d None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:47.961166 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 157/312] 10.222.0.85 () {58 vars in 1334 bytes} [Tue Aug 30 04:12:47 2022] PUT /identity/v3/system/users/db8f383f5f184597955d112061db403c/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:12:47.972110 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d240e13-a824-4d40-8f11-37f68087d62d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:47.978602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-9d240e13-a824-4d40-8f11-37f68087d62d None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:12:47.978602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:47.978602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:47.978602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:47.978602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:47.978602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:47.978602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:47.978602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:47.978602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:47.978602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:47.978602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:47.978602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:47.978602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:47.978602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:47.978602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:47.978602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:47.979550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:47.979550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:47.979550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:47.979550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:47.979550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:47.979550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:47.979550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:47.979550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:47.979550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:47.979550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:47.979550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:47.979550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:47.979550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:47.979550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:47.979550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:47.979550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:47.980340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:47.980340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:47.980340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:47.980340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:47.980340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:47.980340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:47.980340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:47.980340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:47.980340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:47.980340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:47.980340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:12:47.980340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:47.980340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 156/313] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:47 2022] GET /identity/v3/roles/reader => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:12:47.987480 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1be1f6-6257-431e-b6a0-b807c7221c48 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:48.081565 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:48.112291 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1be1f6-6257-431e-b6a0-b807c7221c48 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:48.151648 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f028a0a9-ddb9-4a79-8b79-f1857488dca8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:48.152376 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f028a0a9-ddb9-4a79-8b79-f1857488dca8 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:48.152896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f028a0a9-ddb9-4a79-8b79-f1857488dca8 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:48.153268 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f028a0a9-ddb9-4a79-8b79-f1857488dca8 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:48.154012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f028a0a9-ddb9-4a79-8b79-f1857488dca8 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:48.154384 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f028a0a9-ddb9-4a79-8b79-f1857488dca8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:48.155848 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f028a0a9-ddb9-4a79-8b79-f1857488dca8 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:48.165987 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 158/314] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:47 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:48.171670 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d240e13-a824-4d40-8f11-37f68087d62d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:48.201931 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:48.290190 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d240e13-a824-4d40-8f11-37f68087d62d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:48.329943 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74517ad-1ba3-44b4-96e4-f180fdffb709 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:48.330587 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74517ad-1ba3-44b4-96e4-f180fdffb709 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:48.330867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74517ad-1ba3-44b4-96e4-f180fdffb709 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:48.331143 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74517ad-1ba3-44b4-96e4-f180fdffb709 None admin] PATH_INFO: `/v3/users/dba522ffd9424baf80a496f4095561b7` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:48.332280 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74517ad-1ba3-44b4-96e4-f180fdffb709 None admin] RBAC: Authorizing `identity:get_user(user_id=dba522ffd9424baf80a496f4095561b7)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:48.333886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74517ad-1ba3-44b4-96e4-f180fdffb709 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:48.335785 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 157/315] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:48 2022] GET /identity/v3/users/dba522ffd9424baf80a496f4095561b7 => generated 297 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:48.341290 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f028a0a9-ddb9-4a79-8b79-f1857488dca8 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:48.449838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f028a0a9-ddb9-4a79-8b79-f1857488dca8 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:48.485357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762fc36b-4323-4bb8-9218-defbed436dbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:48.486317 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762fc36b-4323-4bb8-9218-defbed436dbc None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:48.486790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762fc36b-4323-4bb8-9218-defbed436dbc None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:48.487277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762fc36b-4323-4bb8-9218-defbed436dbc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:48.491157 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:48.560406 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762fc36b-4323-4bb8-9218-defbed436dbc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:48.561908 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762fc36b-4323-4bb8-9218-defbed436dbc None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:48.663173 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 159/316] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:12:48.668685 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74517ad-1ba3-44b4-96e4-f180fdffb709 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:48.766407 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:48.776690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74517ad-1ba3-44b4-96e4-f180fdffb709 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:48.815132 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f3fdcd-1a79-4f21-b1e5-93cb32e55609 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:48.816004 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f3fdcd-1a79-4f21-b1e5-93cb32e55609 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:48.816434 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f3fdcd-1a79-4f21-b1e5-93cb32e55609 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:48.816852 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f3fdcd-1a79-4f21-b1e5-93cb32e55609 None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:48.818252 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f3fdcd-1a79-4f21-b1e5-93cb32e55609 None admin] RBAC: Authorizing `identity:get_project(project_id=008c6a0a4a114009bc7c692aabd0b0c8)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:48.819916 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f3fdcd-1a79-4f21-b1e5-93cb32e55609 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:48.822054 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 158/317] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:48 2022] GET /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8 => generated 302 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:48.829060 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762fc36b-4323-4bb8-9218-defbed436dbc None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:48.863750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:48.934173 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762fc36b-4323-4bb8-9218-defbed436dbc None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:48.970144 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ab15f7-76a8-426a-89ae-46910fd96470 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:48.970753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ab15f7-76a8-426a-89ae-46910fd96470 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:48.971023 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ab15f7-76a8-426a-89ae-46910fd96470 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:48.971292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ab15f7-76a8-426a-89ae-46910fd96470 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:48.972008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ab15f7-76a8-426a-89ae-46910fd96470 None admin] RBAC: Adding query filter params (role.id=902fd8d9a4ef48058e1b17e0429ede58, scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8, user.id=dba522ffd9424baf80a496f4095561b7) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:48.972476 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ab15f7-76a8-426a-89ae-46910fd96470 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:48.973933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ab15f7-76a8-426a-89ae-46910fd96470 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:48.980466 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 160/318] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:12:48 2022] GET /identity/v3/role_assignments?user.id=dba522ffd9424baf80a496f4095561b7&scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8&role.id=902fd8d9a4ef48058e1b17e0429ede58 => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:49.389344 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec873bd-ef0c-4128-b15e-505c559f8fd1 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:49.389833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec873bd-ef0c-4128-b15e-505c559f8fd1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:49.389833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec873bd-ef0c-4128-b15e-505c559f8fd1 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:49.391277 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 159/319] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:49.396524 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14dfc45-1d9d-4361-a9e2-92b8838846d8 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:49.396807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14dfc45-1d9d-4361-a9e2-92b8838846d8 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:49.397103 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14dfc45-1d9d-4361-a9e2-92b8838846d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:49.415214 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-c14dfc45-1d9d-4361-a9e2-92b8838846d8 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:49.417856 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c14dfc45-1d9d-4361-a9e2-92b8838846d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:49.483370 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 161/320] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:49.748768 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec873bd-ef0c-4128-b15e-505c559f8fd1 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:49.753477 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7676c334-5efa-4b3c-8d3f-3ed923927261 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:49.753943 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7676c334-5efa-4b3c-8d3f-3ed923927261 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:49.757021 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7676c334-5efa-4b3c-8d3f-3ed923927261 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:49.757021 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 162/321] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:12:49.761670 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e7b664-feff-4e2a-a712-b00639390bd2 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:49.761948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e7b664-feff-4e2a-a712-b00639390bd2 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:49.762495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e7b664-feff-4e2a-a712-b00639390bd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:49.785075 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-63e7b664-feff-4e2a-a712-b00639390bd2 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:49.786308 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-63e7b664-feff-4e2a-a712-b00639390bd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:49.811228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:49.858034 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 163/322] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:49.878837 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec873bd-ef0c-4128-b15e-505c559f8fd1 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:49.912742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991da7b7-b671-4428-816d-b25c0e9df0c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:49.913399 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991da7b7-b671-4428-816d-b25c0e9df0c5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:49.913675 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991da7b7-b671-4428-816d-b25c0e9df0c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:49.913943 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991da7b7-b671-4428-816d-b25c0e9df0c5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:49.915034 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991da7b7-b671-4428-816d-b25c0e9df0c5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:49.916473 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991da7b7-b671-4428-816d-b25c0e9df0c5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:49.918219 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 160/323] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:12:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:49.923557 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e7b664-feff-4e2a-a712-b00639390bd2 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:49.983764 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd646278-3321-497a-82a3-365f0431fe95 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:49.984062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd646278-3321-497a-82a3-365f0431fe95 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:49.984314 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd646278-3321-497a-82a3-365f0431fe95 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:49.985812 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 161/324] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:12:49.991000 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a582fb-a204-45aa-93bb-e2d191fdce93 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:49.991284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a582fb-a204-45aa-93bb-e2d191fdce93 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:49.991555 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a582fb-a204-45aa-93bb-e2d191fdce93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:50.009495 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-30a582fb-a204-45aa-93bb-e2d191fdce93 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:50.010616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-30a582fb-a204-45aa-93bb-e2d191fdce93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:50.027012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e7b664-feff-4e2a-a712-b00639390bd2 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:50.034483 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:50.067932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e10f04-b9ca-4642-b9bc-bdb6e4981bde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:50.069025 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e10f04-b9ca-4642-b9bc-bdb6e4981bde None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:50.069335 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e10f04-b9ca-4642-b9bc-bdb6e4981bde None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:50.069605 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e10f04-b9ca-4642-b9bc-bdb6e4981bde None admin] PATH_INFO: `/v3/groups` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:50.070241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e10f04-b9ca-4642-b9bc-bdb6e4981bde None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:50.071918 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e10f04-b9ca-4642-b9bc-bdb6e4981bde None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:50.091484 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 162/325] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:50.093187 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 164/326] 10.222.0.85 () {60 vars in 1134 bytes} [Tue Aug 30 04:12:49 2022] POST /identity/v3/groups => generated 226 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:12:50.125023 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a582fb-a204-45aa-93bb-e2d191fdce93 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:50.233542 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a582fb-a204-45aa-93bb-e2d191fdce93 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:50.270383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5827cd8d-c2e7-49bc-bde1-ae42007297f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:50.270943 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5827cd8d-c2e7-49bc-bde1-ae42007297f6 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:50.271247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5827cd8d-c2e7-49bc-bde1-ae42007297f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:50.271520 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5827cd8d-c2e7-49bc-bde1-ae42007297f6 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:50.283991 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5827cd8d-c2e7-49bc-bde1-ae42007297f6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:50.289065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-5827cd8d-c2e7-49bc-bde1-ae42007297f6 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:12:50.289065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:50.289065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:50.289065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:50.289065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:50.289065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:50.289065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:50.289065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:50.289065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:50.289065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:50.289065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:50.289065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:50.289065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:50.289065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:50.289065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:50.289065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:50.289948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:50.289948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:50.289948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:50.289948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:50.289948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:50.289948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:50.289948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:50.289948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:50.289948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:50.289948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:50.289948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:50.289948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:50.289948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:50.289948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:50.289948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:50.289948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:50.290901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:50.290901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:50.290901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:50.290901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:50.290901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:50.290901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:50.290901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:50.290901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:50.290901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:50.290901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:50.290901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:12:50.290901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:50.290901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 163/327] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:50 2022] GET /identity/v3/roles/reader => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:50.296022 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e10f04-b9ca-4642-b9bc-bdb6e4981bde None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:50.320147 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:50.378695 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5827cd8d-c2e7-49bc-bde1-ae42007297f6 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:50.404841 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:50.416075 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e10f04-b9ca-4642-b9bc-bdb6e4981bde None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:50.461569 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3d1b34-3a3f-44ed-9eb5-4af31435b065 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:50.462436 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3d1b34-3a3f-44ed-9eb5-4af31435b065 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:50.462715 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3d1b34-3a3f-44ed-9eb5-4af31435b065 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:50.462989 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3d1b34-3a3f-44ed-9eb5-4af31435b065 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:50.464266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3d1b34-3a3f-44ed-9eb5-4af31435b065 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:50.464266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3d1b34-3a3f-44ed-9eb5-4af31435b065 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:50.465372 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3d1b34-3a3f-44ed-9eb5-4af31435b065 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:50.475238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 165/328] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:50 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:50.481483 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3d1b34-3a3f-44ed-9eb5-4af31435b065 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:50.516659 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5827cd8d-c2e7-49bc-bde1-ae42007297f6 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:50.569680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9782fd-6b53-4257-a21a-d3717f1186ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:50.573829 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9782fd-6b53-4257-a21a-d3717f1186ea None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:50.574118 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9782fd-6b53-4257-a21a-d3717f1186ea None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:50.574399 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9782fd-6b53-4257-a21a-d3717f1186ea None admin] PATH_INFO: `/v3/roles/member` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:50.584950 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:50.591373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9782fd-6b53-4257-a21a-d3717f1186ea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:50.596896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-8c9782fd-6b53-4257-a21a-d3717f1186ea None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:50.596896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:50.596896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:50.596896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:50.596896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:50.596896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:50.596896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:50.596896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:50.596896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:50.596896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:50.596896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:50.596896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:50.596896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:50.596896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:50.596896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:50.596896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:50.597853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:50.597853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:50.597853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:50.597853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:50.597853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:50.597853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:50.597853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:50.597853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:50.597853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:50.597853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:50.597853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:50.597853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:50.597853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:50.597853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:50.597853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:50.597853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:50.599562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:50.599562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:50.599562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:50.599562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:50.599562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:50.599562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:50.599562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:50.599562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:50.599562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:50.599562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:50.599562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:50.599562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:50.599562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 164/329] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:50 2022] GET /identity/v3/roles/member => generated 84 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:12:50.603868 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9782fd-6b53-4257-a21a-d3717f1186ea None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:50.612151 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3d1b34-3a3f-44ed-9eb5-4af31435b065 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:50.659028 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38655383-03c9-438b-8b8d-dbb61385aad7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:50.659784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38655383-03c9-438b-8b8d-dbb61385aad7 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:50.659984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38655383-03c9-438b-8b8d-dbb61385aad7 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:50.660227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38655383-03c9-438b-8b8d-dbb61385aad7 None admin] PATH_INFO: `/v3/users/e8d54ce3bc0746c2a48bbda5024200b7` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:50.662320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38655383-03c9-438b-8b8d-dbb61385aad7 None admin] RBAC: Authorizing `identity:get_user(user_id=e8d54ce3bc0746c2a48bbda5024200b7)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:50.663772 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38655383-03c9-438b-8b8d-dbb61385aad7 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:50.665749 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 166/330] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:50 2022] GET /identity/v3/users/e8d54ce3bc0746c2a48bbda5024200b7 => generated 293 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:50.671130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38655383-03c9-438b-8b8d-dbb61385aad7 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:50.720218 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:50.752564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9782fd-6b53-4257-a21a-d3717f1186ea None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:50.788214 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38655383-03c9-438b-8b8d-dbb61385aad7 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:50.802388 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e6fd6f-fc59-4bca-a8eb-980468b39300 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:50.803627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e6fd6f-fc59-4bca-a8eb-980468b39300 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:50.804563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e6fd6f-fc59-4bca-a8eb-980468b39300 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:50.805179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e6fd6f-fc59-4bca-a8eb-980468b39300 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:50.806157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e6fd6f-fc59-4bca-a8eb-980468b39300 None admin] RBAC: Adding query filter params (name=member) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:50.806892 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e6fd6f-fc59-4bca-a8eb-980468b39300 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:50.808515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e6fd6f-fc59-4bca-a8eb-980468b39300 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:50.821575 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 165/331] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:50 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:50.827559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e6fd6f-fc59-4bca-a8eb-980468b39300 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:50.836848 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7531767f-edce-4d54-8c2b-b37ff1c593e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:50.837541 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7531767f-edce-4d54-8c2b-b37ff1c593e4 None admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:50.837822 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7531767f-edce-4d54-8c2b-b37ff1c593e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:50.838117 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7531767f-edce-4d54-8c2b-b37ff1c593e4 None admin] PATH_INFO: `/v3/system/users/e8d54ce3bc0746c2a48bbda5024200b7/roles/902fd8d9a4ef48058e1b17e0429ede58` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:50.839586 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7531767f-edce-4d54-8c2b-b37ff1c593e4 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=e8d54ce3bc0746c2a48bbda5024200b7, role_id=902fd8d9a4ef48058e1b17e0429ede58)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:50.841028 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7531767f-edce-4d54-8c2b-b37ff1c593e4 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:50.853827 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 167/332] 10.222.0.85 () {58 vars in 1334 bytes} [Tue Aug 30 04:12:50 2022] PUT /identity/v3/system/users/e8d54ce3bc0746c2a48bbda5024200b7/roles/902fd8d9a4ef48058e1b17e0429ede58 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:12:50.968839 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e6fd6f-fc59-4bca-a8eb-980468b39300 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:51.006643 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229c67e1-ae03-438a-aa98-52404eadb9b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:51.007305 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229c67e1-ae03-438a-aa98-52404eadb9b5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:51.007577 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229c67e1-ae03-438a-aa98-52404eadb9b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:51.007853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229c67e1-ae03-438a-aa98-52404eadb9b5 None admin] PATH_INFO: `/v3/users/db8f383f5f184597955d112061db403c` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:51.009025 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229c67e1-ae03-438a-aa98-52404eadb9b5 None admin] RBAC: Authorizing `identity:get_user(user_id=db8f383f5f184597955d112061db403c)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:51.010553 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229c67e1-ae03-438a-aa98-52404eadb9b5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:51.012470 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 166/333] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:50 2022] GET /identity/v3/users/db8f383f5f184597955d112061db403c => generated 293 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:51.018810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7531767f-edce-4d54-8c2b-b37ff1c593e4 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:51.037713 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:51.141647 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7531767f-edce-4d54-8c2b-b37ff1c593e4 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:51.189308 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb3586c-6337-4277-8ecb-d0457bfe8275 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:51.189932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb3586c-6337-4277-8ecb-d0457bfe8275 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:51.190215 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb3586c-6337-4277-8ecb-d0457bfe8275 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:51.190489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb3586c-6337-4277-8ecb-d0457bfe8275 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:51.191257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb3586c-6337-4277-8ecb-d0457bfe8275 None admin] RBAC: Adding query filter params (role.id=ae0cef8120cf4dfc899c30803243a851, user.id=db8f383f5f184597955d112061db403c, scope.system=all) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:51.191675 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb3586c-6337-4277-8ecb-d0457bfe8275 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:51.193181 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb3586c-6337-4277-8ecb-d0457bfe8275 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:51.211233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 168/334] 10.222.0.85 () {56 vars in 1308 bytes} [Tue Aug 30 04:12:51 2022] GET /identity/v3/role_assignments?user.id=db8f383f5f184597955d112061db403c&scope.system=all&role.id=ae0cef8120cf4dfc899c30803243a851 => generated 510 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:52.641551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f13f57c-49b3-4e54-aad1-2ce80adb8049 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:52.642612 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f13f57c-49b3-4e54-aad1-2ce80adb8049 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:52.643066 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f13f57c-49b3-4e54-aad1-2ce80adb8049 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:52.644992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 167/335] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:52.651516 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d3b867-f3bb-4471-a386-7aec180391f0 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:52.651954 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d3b867-f3bb-4471-a386-7aec180391f0 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:52.652135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d3b867-f3bb-4471-a386-7aec180391f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:52.678110 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-73d3b867-f3bb-4471-a386-7aec180391f0 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:52.679313 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-73d3b867-f3bb-4471-a386-7aec180391f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:52.684230 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0609f0-eabd-4086-9bfe-9888a225a475 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:52.684756 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0609f0-eabd-4086-9bfe-9888a225a475 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:52.685259 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0609f0-eabd-4086-9bfe-9888a225a475 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:52.686807 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 168/336] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:52.696368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce2efe2-a01f-45bf-a142-28e5c343aba6 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:52.697163 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce2efe2-a01f-45bf-a142-28e5c343aba6 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:52.697839 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce2efe2-a01f-45bf-a142-28e5c343aba6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:52.727095 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3ce2efe2-a01f-45bf-a142-28e5c343aba6 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:52.728633 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3ce2efe2-a01f-45bf-a142-28e5c343aba6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:52.732028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:52.831700 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 169/337] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:52.852244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 169/338] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:53.186925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e392370-405f-4738-8791-473d45128574 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:53.187255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e392370-405f-4738-8791-473d45128574 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:53.187479 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e392370-405f-4738-8791-473d45128574 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:53.189213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 170/339] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:53.194524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10870e8b-bd28-4738-b8f0-53ad7a1b13d6 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:53.195052 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10870e8b-bd28-4738-b8f0-53ad7a1b13d6 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:53.195534 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10870e8b-bd28-4738-b8f0-53ad7a1b13d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:53.222103 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-10870e8b-bd28-4738-b8f0-53ad7a1b13d6 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:53.223727 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e392370-405f-4738-8791-473d45128574 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:53.223965 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-10870e8b-bd28-4738-b8f0-53ad7a1b13d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:53.227659 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:53.360529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 170/340] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:53.361726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10870e8b-bd28-4738-b8f0-53ad7a1b13d6 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:53.378056 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e392370-405f-4738-8791-473d45128574 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:53.423142 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbf642e-aee0-400c-9acc-72f7d3a53030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:53.423789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbf642e-aee0-400c-9acc-72f7d3a53030 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:53.424068 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbf642e-aee0-400c-9acc-72f7d3a53030 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:53.424340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbf642e-aee0-400c-9acc-72f7d3a53030 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:53.447118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbf642e-aee0-400c-9acc-72f7d3a53030 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:53.452232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-7fbf642e-aee0-400c-9acc-72f7d3a53030 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:12:53.452232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:53.452232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:53.452232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:53.452232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:53.452232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:53.452232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:53.452232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:53.452232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:53.452232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:53.452232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:53.452232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:53.452232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:53.452232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:53.452232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:53.452232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:53.453112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:53.453112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:53.453112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:53.453112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:53.453112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:53.453112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:53.453112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:53.453112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:53.453112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:53.453112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:53.453112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:53.453112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:53.453112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:53.453112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:53.453112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:53.453112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:53.453907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:53.453907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:53.453907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:53.453907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:53.453907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:53.453907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:53.453907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:53.453907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:53.453907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:53.453907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:53.453907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:12:53.453907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:53.457780 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 171/341] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:12:53 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:53.463233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbf642e-aee0-400c-9acc-72f7d3a53030 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:53.480242 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:53.518261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10870e8b-bd28-4738-b8f0-53ad7a1b13d6 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:53.565067 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:53.586708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac2f660-4f69-415e-b271-2539509fc0d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:53.587722 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac2f660-4f69-415e-b271-2539509fc0d6 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:53.588202 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac2f660-4f69-415e-b271-2539509fc0d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:53.588674 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac2f660-4f69-415e-b271-2539509fc0d6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:53.606022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac2f660-4f69-415e-b271-2539509fc0d6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:53.611334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-bac2f660-4f69-415e-b271-2539509fc0d6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:53.611334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:53.611334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:53.611334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:53.611334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:53.611334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:53.611334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:53.611334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:53.611334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:53.611334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:53.611334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:53.611334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:53.611334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:53.611334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:53.611334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:53.611334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:53.612178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:53.612178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:53.612178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:53.612178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:53.612178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:53.612178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:53.612178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:53.612178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:53.612178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:53.612178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:53.612178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:53.612178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:53.612178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:53.612178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:53.612178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:53.612178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:53.612970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:53.612970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:53.612970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:53.612970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:53.612970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:53.612970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:53.612970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:53.612970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:53.612970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:53.612970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:53.612970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:53.612970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:53.619175 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 171/342] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:53.625088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac2f660-4f69-415e-b271-2539509fc0d6 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:53.626218 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbf642e-aee0-400c-9acc-72f7d3a53030 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:53.691035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58c7f7e-8201-45f3-8367-b4242dcbf038 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:53.691661 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58c7f7e-8201-45f3-8367-b4242dcbf038 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:53.691823 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58c7f7e-8201-45f3-8367-b4242dcbf038 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:53.692102 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58c7f7e-8201-45f3-8367-b4242dcbf038 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:53.692789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58c7f7e-8201-45f3-8367-b4242dcbf038 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:53.693149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58c7f7e-8201-45f3-8367-b4242dcbf038 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:53.694548 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58c7f7e-8201-45f3-8367-b4242dcbf038 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:53.707324 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 172/343] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:12:53 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:53.715564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58c7f7e-8201-45f3-8367-b4242dcbf038 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:53.810571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac2f660-4f69-415e-b271-2539509fc0d6 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:53.866094 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58c7f7e-8201-45f3-8367-b4242dcbf038 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:53.885241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:53.900215 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:53.909733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab26fed4-cef4-48ff-bbaf-6b702f7afc92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:53.910405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab26fed4-cef4-48ff-bbaf-6b702f7afc92 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:53.910793 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab26fed4-cef4-48ff-bbaf-6b702f7afc92 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:53.911149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab26fed4-cef4-48ff-bbaf-6b702f7afc92 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:53.911795 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab26fed4-cef4-48ff-bbaf-6b702f7afc92 None admin] RBAC: Adding query filter params (name=member) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:53.912205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab26fed4-cef4-48ff-bbaf-6b702f7afc92 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:53.913764 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab26fed4-cef4-48ff-bbaf-6b702f7afc92 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:53.915821 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcc5c20-03b6-4ab0-9bc6-5ceb68212a28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:53.916452 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcc5c20-03b6-4ab0-9bc6-5ceb68212a28 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:53.916734 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcc5c20-03b6-4ab0-9bc6-5ceb68212a28 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:53.917011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcc5c20-03b6-4ab0-9bc6-5ceb68212a28 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:53.927979 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 172/344] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:53 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:53.928851 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab26fed4-cef4-48ff-bbaf-6b702f7afc92 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:53.996845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcc5c20-03b6-4ab0-9bc6-5ceb68212a28 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:53.998373 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcc5c20-03b6-4ab0-9bc6-5ceb68212a28 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:54.117753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 173/345] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:12:54.117753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcc5c20-03b6-4ab0-9bc6-5ceb68212a28 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:54.144668 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:54.157566 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab26fed4-cef4-48ff-bbaf-6b702f7afc92 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:54.197543 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b164f81b-e3b1-4548-be3c-f020caa8d105 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:54.198183 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b164f81b-e3b1-4548-be3c-f020caa8d105 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:54.198460 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b164f81b-e3b1-4548-be3c-f020caa8d105 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:54.198731 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b164f81b-e3b1-4548-be3c-f020caa8d105 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:54.211590 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b164f81b-e3b1-4548-be3c-f020caa8d105 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:54.217284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-b164f81b-e3b1-4548-be3c-f020caa8d105 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:54.217284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:54.217284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:54.217284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:54.217284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:54.217284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:54.217284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:54.217284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:54.217284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:54.217284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:54.217284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:54.217284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:54.217284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:54.217284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:54.217284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:54.217284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:54.218168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:54.218168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:54.218168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:54.218168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:54.218168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:54.218168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:54.218168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:54.218168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:54.218168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:54.218168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:54.218168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:54.218168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:54.218168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:54.218168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:54.218168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:54.218168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:54.219069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:54.219069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:54.219069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:54.219069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:54.219069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:54.219069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:54.219069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:54.219069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:54.219069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:54.219069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:54.219069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:54.219069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:54.223575 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 173/346] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:54.225254 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7421ed2-ccde-4935-b4fc-744d850d0724 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:54.225718 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7421ed2-ccde-4935-b4fc-744d850d0724 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:54.226211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7421ed2-ccde-4935-b4fc-744d850d0724 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:54.228141 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 174/347] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:54.229000 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7421ed2-ccde-4935-b4fc-744d850d0724 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:54.237676 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcc5c20-03b6-4ab0-9bc6-5ceb68212a28 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:54.291216 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100e58e5-8d21-4599-b132-7bf43c6d15e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:54.291822 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100e58e5-8d21-4599-b132-7bf43c6d15e5 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:54.292873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100e58e5-8d21-4599-b132-7bf43c6d15e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:54.293359 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100e58e5-8d21-4599-b132-7bf43c6d15e5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:54.296859 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:54.356972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7421ed2-ccde-4935-b4fc-744d850d0724 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:54.384327 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100e58e5-8d21-4599-b132-7bf43c6d15e5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:54.386084 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100e58e5-8d21-4599-b132-7bf43c6d15e5 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:54.404220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e11a66-10ee-4321-b7cf-758e6b8dec3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:54.405279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e11a66-10ee-4321-b7cf-758e6b8dec3b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:54.405762 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e11a66-10ee-4321-b7cf-758e6b8dec3b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:54.406252 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e11a66-10ee-4321-b7cf-758e6b8dec3b None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:54.407619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e11a66-10ee-4321-b7cf-758e6b8dec3b None admin] RBAC: Authorizing `identity:get_project(project_id=26661cf5060b4b939d8ab02f9ea45daf)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:54.409364 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e11a66-10ee-4321-b7cf-758e6b8dec3b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:54.412169 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 175/348] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:54 2022] GET /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf => generated 298 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:54.413612 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88941078-e056-4bac-bf61-49325b85436f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:54.414128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88941078-e056-4bac-bf61-49325b85436f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:54.414616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88941078-e056-4bac-bf61-49325b85436f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:54.417224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:54.434995 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-88941078-e056-4bac-bf61-49325b85436f None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:54.436412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-88941078-e056-4bac-bf61-49325b85436f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:54.504556 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 174/349] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:12:54.505473 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100e58e5-8d21-4599-b132-7bf43c6d15e5 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:54.525844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 176/350] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:54.527384 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b537f91-b6ee-425e-86af-e3b23b6d3b8c None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:54.527811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b537f91-b6ee-425e-86af-e3b23b6d3b8c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:54.528226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b537f91-b6ee-425e-86af-e3b23b6d3b8c None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:54.529898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 177/351] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:54.530752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b537f91-b6ee-425e-86af-e3b23b6d3b8c None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:54.591771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:54.618367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100e58e5-8d21-4599-b132-7bf43c6d15e5 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:54.647115 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:54.659777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1039f7df-f236-4a48-8061-a2b65664d7f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:54.660717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1039f7df-f236-4a48-8061-a2b65664d7f5 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:54.661235 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1039f7df-f236-4a48-8061-a2b65664d7f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:54.661742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1039f7df-f236-4a48-8061-a2b65664d7f5 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:54.663255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1039f7df-f236-4a48-8061-a2b65664d7f5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:54.663778 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1039f7df-f236-4a48-8061-a2b65664d7f5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:54.665480 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1039f7df-f236-4a48-8061-a2b65664d7f5 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:54.666290 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b537f91-b6ee-425e-86af-e3b23b6d3b8c None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:54.675103 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 175/352] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:54.676315 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1039f7df-f236-4a48-8061-a2b65664d7f5 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:54.722299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49698b8d-38bd-4f6b-8013-75d4a4617c74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:54.723208 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49698b8d-38bd-4f6b-8013-75d4a4617c74 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:54.723625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49698b8d-38bd-4f6b-8013-75d4a4617c74 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:54.724031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49698b8d-38bd-4f6b-8013-75d4a4617c74 None admin] PATH_INFO: `/v3/groups/329d8769e54e48bc8acd9188320c782a` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:54.736835 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49698b8d-38bd-4f6b-8013-75d4a4617c74 None admin] RBAC: Authorizing `identity:get_group(group_id=329d8769e54e48bc8acd9188320c782a)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:54.738615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49698b8d-38bd-4f6b-8013-75d4a4617c74 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:54.741550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 178/353] 10.222.0.85 () {56 vars in 1180 bytes} [Tue Aug 30 04:12:54 2022] GET /identity/v3/groups/329d8769e54e48bc8acd9188320c782a => generated 226 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:54.743335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a00357-5c22-4a23-967e-f1f4da92d587 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:54.743757 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a00357-5c22-4a23-967e-f1f4da92d587 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:54.744171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a00357-5c22-4a23-967e-f1f4da92d587 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:54.768258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-a4a00357-5c22-4a23-967e-f1f4da92d587 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:54.769723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a4a00357-5c22-4a23-967e-f1f4da92d587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:54.792656 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1039f7df-f236-4a48-8061-a2b65664d7f5 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:54.833232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9de5816-f02f-4df6-a969-028b61cbbf72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:54.834710 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9de5816-f02f-4df6-a969-028b61cbbf72 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:54.835172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9de5816-f02f-4df6-a969-028b61cbbf72 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:54.835647 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9de5816-f02f-4df6-a969-028b61cbbf72 None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:54.837193 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9de5816-f02f-4df6-a969-028b61cbbf72 None admin] RBAC: Authorizing `identity:get_project(project_id=26661cf5060b4b939d8ab02f9ea45daf)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:54.838822 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9de5816-f02f-4df6-a969-028b61cbbf72 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:54.841194 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 176/354] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:54 2022] GET /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf => generated 298 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:54.842198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9de5816-f02f-4df6-a969-028b61cbbf72 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:54.857796 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 179/355] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:54.859702 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a00357-5c22-4a23-967e-f1f4da92d587 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:54.914340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:54.924060 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:54.969314 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9de5816-f02f-4df6-a969-028b61cbbf72 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:55.005988 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a00357-5c22-4a23-967e-f1f4da92d587 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:55.006470 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45843b94-f587-426b-8bd7-c85938ab41f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:55.007131 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45843b94-f587-426b-8bd7-c85938ab41f9 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:55.007476 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45843b94-f587-426b-8bd7-c85938ab41f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:55.007812 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45843b94-f587-426b-8bd7-c85938ab41f9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:55.044964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e9e916-80ac-4b42-8acd-061ced3280c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:55.045928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e9e916-80ac-4b42-8acd-061ced3280c5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:55.046443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e9e916-80ac-4b42-8acd-061ced3280c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:55.046916 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e9e916-80ac-4b42-8acd-061ced3280c5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:55.047850 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e9e916-80ac-4b42-8acd-061ced3280c5 None admin] RBAC: Adding query filter params (group.id=329d8769e54e48bc8acd9188320c782a, role.id=a9e862ac528245ca8363bccc5bf35287, scope.project.id=26661cf5060b4b939d8ab02f9ea45daf) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:55.048361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e9e916-80ac-4b42-8acd-061ced3280c5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:55.050315 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e9e916-80ac-4b42-8acd-061ced3280c5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:55.057241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 180/356] 10.222.0.85 () {56 vars in 1376 bytes} [Tue Aug 30 04:12:54 2022] GET /identity/v3/role_assignments?group.id=329d8769e54e48bc8acd9188320c782a&scope.project.id=26661cf5060b4b939d8ab02f9ea45daf&role.id=a9e862ac528245ca8363bccc5bf35287 => generated 260 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:55.057947 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e9e916-80ac-4b42-8acd-061ced3280c5 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:55.097287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45843b94-f587-426b-8bd7-c85938ab41f9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:55.099125 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45843b94-f587-426b-8bd7-c85938ab41f9 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:55.185907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:55.202762 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e9e916-80ac-4b42-8acd-061ced3280c5 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:55.212462 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:55.247544 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 177/357] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:12:55.249958 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45843b94-f587-426b-8bd7-c85938ab41f9 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:55.253750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc95188-f98d-4115-88cf-cdecff9d0def None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:55.254594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc95188-f98d-4115-88cf-cdecff9d0def None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:55.255200 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc95188-f98d-4115-88cf-cdecff9d0def None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:55.255480 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc95188-f98d-4115-88cf-cdecff9d0def None admin] PATH_INFO: `/v3/groups/329d8769e54e48bc8acd9188320c782a` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:55.256892 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc95188-f98d-4115-88cf-cdecff9d0def None admin] RBAC: Authorizing `identity:get_group(group_id=329d8769e54e48bc8acd9188320c782a)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:55.258384 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc95188-f98d-4115-88cf-cdecff9d0def None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:55.259981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 181/358] 10.222.0.85 () {56 vars in 1180 bytes} [Tue Aug 30 04:12:55 2022] GET /identity/v3/groups/329d8769e54e48bc8acd9188320c782a => generated 226 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:55.261438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac364bd-8352-48be-a9df-9be96bd8b8e3 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:55.261730 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac364bd-8352-48be-a9df-9be96bd8b8e3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:55.262083 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac364bd-8352-48be-a9df-9be96bd8b8e3 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:55.263492 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 182/359] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:55.264418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac364bd-8352-48be-a9df-9be96bd8b8e3 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:55.377718 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45843b94-f587-426b-8bd7-c85938ab41f9 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:55.392166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac364bd-8352-48be-a9df-9be96bd8b8e3 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:55.418965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4145b7a9-38fa-4716-897e-3f7e1a09dfe4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:55.419884 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4145b7a9-38fa-4716-897e-3f7e1a09dfe4 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:55.420306 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4145b7a9-38fa-4716-897e-3f7e1a09dfe4 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:55.420719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4145b7a9-38fa-4716-897e-3f7e1a09dfe4 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:55.434514 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4145b7a9-38fa-4716-897e-3f7e1a09dfe4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:55.436938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85b5a10-c05e-492a-ac7e-230b2c57cb5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:55.437566 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85b5a10-c05e-492a-ac7e-230b2c57cb5a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:55.437847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85b5a10-c05e-492a-ac7e-230b2c57cb5a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:55.438123 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85b5a10-c05e-492a-ac7e-230b2c57cb5a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:55.439746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-4145b7a9-38fa-4716-897e-3f7e1a09dfe4 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:12:55.439746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:55.439746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:55.439746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:55.439746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:55.439746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:55.439746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:55.439746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:55.439746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:55.439746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:55.439746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:55.439746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:55.439746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:55.439746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:55.439746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:55.439746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:55.440634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:55.440634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:55.440634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:55.440634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:55.440634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:55.440634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:55.440634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:55.440634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:55.440634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:55.440634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:55.440634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:55.440634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:55.440634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:55.440634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:55.440634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:55.440634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:55.441472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:55.441472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:55.441472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:55.441472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:55.441472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:55.441472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:55.441472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:55.441472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:55.441472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:55.441472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:55.441472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 04:12:55.441472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:55.441472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 178/360] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:55 2022] GET /identity/v3/roles/reader => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:55.442215 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4145b7a9-38fa-4716-897e-3f7e1a09dfe4 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:55.449977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85b5a10-c05e-492a-ac7e-230b2c57cb5a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:55.455037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-b85b5a10-c05e-492a-ac7e-230b2c57cb5a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:12:55.455037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:55.455037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:55.455037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:55.455037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:55.455037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:55.455037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:55.455037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:55.455037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:55.455037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:55.455037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:55.455037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:55.455037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:55.455037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:55.455037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:55.455037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:55.455950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:55.455950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:55.455950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:55.455950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:55.455950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:55.455950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:55.455950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:55.455950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:55.455950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:55.455950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:55.455950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:55.455950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:55.455950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:55.455950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:55.455950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:55.455950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:55.456763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:55.456763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:55.456763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:55.456763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:55.456763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:55.456763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:55.456763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:55.456763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:55.456763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:55.456763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:55.456763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:12:55.456763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:12:55.456763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 183/361] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:12:55 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:55.457474 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85b5a10-c05e-492a-ac7e-230b2c57cb5a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:55.465995 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:55.498035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:55.578888 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4145b7a9-38fa-4716-897e-3f7e1a09dfe4 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:55.590877 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85b5a10-c05e-492a-ac7e-230b2c57cb5a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:55.621199 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6912074c-872f-4ecb-a664-3186fd294029 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:55.621996 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6912074c-872f-4ecb-a664-3186fd294029 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:55.622422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6912074c-872f-4ecb-a664-3186fd294029 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:55.622792 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6912074c-872f-4ecb-a664-3186fd294029 None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:55.625105 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6912074c-872f-4ecb-a664-3186fd294029 None admin] RBAC: Authorizing `identity:get_project(project_id=008c6a0a4a114009bc7c692aabd0b0c8)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:55.627165 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6912074c-872f-4ecb-a664-3186fd294029 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:55.628736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 179/362] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:55 2022] GET /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8 => generated 302 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:55.629873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d284ce21-b65a-49cf-9a2c-52799d5e7ead None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:55.630089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d284ce21-b65a-49cf-9a2c-52799d5e7ead None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:55.630389 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d284ce21-b65a-49cf-9a2c-52799d5e7ead None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:55.643181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63f87b4-bc91-4a25-8351-1443749d2d77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:55.644837 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63f87b4-bc91-4a25-8351-1443749d2d77 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:55.647503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63f87b4-bc91-4a25-8351-1443749d2d77 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:55.647503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63f87b4-bc91-4a25-8351-1443749d2d77 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:55.647503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63f87b4-bc91-4a25-8351-1443749d2d77 None admin] RBAC: Adding query filter params (group.id=329d8769e54e48bc8acd9188320c782a, role.id=ae0cef8120cf4dfc899c30803243a851, scope.project.id=26661cf5060b4b939d8ab02f9ea45daf) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:55.647503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63f87b4-bc91-4a25-8351-1443749d2d77 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:55.648342 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63f87b4-bc91-4a25-8351-1443749d2d77 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:55.655580 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 184/363] 10.222.0.85 () {56 vars in 1376 bytes} [Tue Aug 30 04:12:55 2022] GET /identity/v3/role_assignments?group.id=329d8769e54e48bc8acd9188320c782a&scope.project.id=26661cf5060b4b939d8ab02f9ea45daf&role.id=ae0cef8120cf4dfc899c30803243a851 => generated 260 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:55.656573 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63f87b4-bc91-4a25-8351-1443749d2d77 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:55.656830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-d284ce21-b65a-49cf-9a2c-52799d5e7ead None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:55.658113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d284ce21-b65a-49cf-9a2c-52799d5e7ead None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:55.751246 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 180/364] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:55.752076 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d284ce21-b65a-49cf-9a2c-52799d5e7ead None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:55.776577 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:55.777880 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:55.785476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63f87b4-bc91-4a25-8351-1443749d2d77 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:55.855557 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a54f5a3-2ee5-4d27-b7ec-cdd643d799d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:55.856497 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a54f5a3-2ee5-4d27-b7ec-cdd643d799d5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:55.856954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a54f5a3-2ee5-4d27-b7ec-cdd643d799d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:55.857529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a54f5a3-2ee5-4d27-b7ec-cdd643d799d5 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:55.858371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a54f5a3-2ee5-4d27-b7ec-cdd643d799d5 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:55.859277 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a54f5a3-2ee5-4d27-b7ec-cdd643d799d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:55.860835 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a54f5a3-2ee5-4d27-b7ec-cdd643d799d5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:55.871364 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 185/365] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:55 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:55.872647 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a54f5a3-2ee5-4d27-b7ec-cdd643d799d5 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:55.913853 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d284ce21-b65a-49cf-9a2c-52799d5e7ead None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:55.965734 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c359ad1c-5409-48a5-9491-45e668ef658f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:55.966496 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c359ad1c-5409-48a5-9491-45e668ef658f None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:55.966847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c359ad1c-5409-48a5-9491-45e668ef658f None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:55.967191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c359ad1c-5409-48a5-9491-45e668ef658f None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:55.967937 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c359ad1c-5409-48a5-9491-45e668ef658f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:55.968313 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c359ad1c-5409-48a5-9491-45e668ef658f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:55.971861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c359ad1c-5409-48a5-9491-45e668ef658f None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:55.982766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 181/366] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:12:55 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:55.984120 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c359ad1c-5409-48a5-9491-45e668ef658f None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:56.013742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a54f5a3-2ee5-4d27-b7ec-cdd643d799d5 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:56.080972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb6f382-d71b-47c4-b78a-d985280cc44b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:56.082762 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb6f382-d71b-47c4-b78a-d985280cc44b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:56.083384 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb6f382-d71b-47c4-b78a-d985280cc44b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:56.085149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb6f382-d71b-47c4-b78a-d985280cc44b None admin] PATH_INFO: `/v3/groups/329d8769e54e48bc8acd9188320c782a` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:56.086697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb6f382-d71b-47c4-b78a-d985280cc44b None admin] RBAC: Authorizing `identity:get_group(group_id=329d8769e54e48bc8acd9188320c782a)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:56.090423 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb6f382-d71b-47c4-b78a-d985280cc44b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:56.093365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 186/367] 10.222.0.85 () {56 vars in 1180 bytes} [Tue Aug 30 04:12:55 2022] GET /identity/v3/groups/329d8769e54e48bc8acd9188320c782a => generated 226 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:56.095561 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb6f382-d71b-47c4-b78a-d985280cc44b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:56.139240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:56.152208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:56.204725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c359ad1c-5409-48a5-9491-45e668ef658f None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:56.252804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50d1d03-f7e8-4772-8df1-493694e920ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:56.253484 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50d1d03-f7e8-4772-8df1-493694e920ab None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:56.253765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50d1d03-f7e8-4772-8df1-493694e920ab None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:56.254041 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50d1d03-f7e8-4772-8df1-493694e920ab None admin] PATH_INFO: `/v3/users/e8d54ce3bc0746c2a48bbda5024200b7` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:56.256015 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50d1d03-f7e8-4772-8df1-493694e920ab None admin] RBAC: Authorizing `identity:get_user(user_id=e8d54ce3bc0746c2a48bbda5024200b7)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:56.258776 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50d1d03-f7e8-4772-8df1-493694e920ab None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:56.259340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb6f382-d71b-47c4-b78a-d985280cc44b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:56.260801 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 182/368] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:12:55 2022] GET /identity/v3/users/e8d54ce3bc0746c2a48bbda5024200b7 => generated 293 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:56.262072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50d1d03-f7e8-4772-8df1-493694e920ab None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:56.300348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a789cf-1dde-4ad9-a877-4facd7d8ab8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:56.301912 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a789cf-1dde-4ad9-a877-4facd7d8ab8b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:56.302712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a789cf-1dde-4ad9-a877-4facd7d8ab8b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:56.303004 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a789cf-1dde-4ad9-a877-4facd7d8ab8b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:56.396196 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50d1d03-f7e8-4772-8df1-493694e920ab None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:56.398369 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a789cf-1dde-4ad9-a877-4facd7d8ab8b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:56.399856 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a789cf-1dde-4ad9-a877-4facd7d8ab8b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:56.435347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:56.439004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98fdde2-3c89-41fc-8f82-e4e41844de34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:56.439630 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98fdde2-3c89-41fc-8f82-e4e41844de34 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:56.439910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98fdde2-3c89-41fc-8f82-e4e41844de34 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:56.440182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98fdde2-3c89-41fc-8f82-e4e41844de34 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:56.440927 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98fdde2-3c89-41fc-8f82-e4e41844de34 None admin] RBAC: Adding query filter params (group.id=329d8769e54e48bc8acd9188320c782a, role.id=ae0cef8120cf4dfc899c30803243a851, scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:56.441270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98fdde2-3c89-41fc-8f82-e4e41844de34 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:56.442719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98fdde2-3c89-41fc-8f82-e4e41844de34 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:56.457395 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 183/369] 10.222.0.85 () {56 vars in 1376 bytes} [Tue Aug 30 04:12:56 2022] GET /identity/v3/role_assignments?group.id=329d8769e54e48bc8acd9188320c782a&scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8&role.id=ae0cef8120cf4dfc899c30803243a851 => generated 260 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:56.458359 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98fdde2-3c89-41fc-8f82-e4e41844de34 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:56.476425 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:56.577099 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 187/370] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 481 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:12:56.577927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a789cf-1dde-4ad9-a877-4facd7d8ab8b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:56.654969 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98fdde2-3c89-41fc-8f82-e4e41844de34 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:56.717309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bca1158-514e-4830-8126-c1680c6f8167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:56.717948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bca1158-514e-4830-8126-c1680c6f8167 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:56.718222 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bca1158-514e-4830-8126-c1680c6f8167 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:56.718487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bca1158-514e-4830-8126-c1680c6f8167 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:56.730529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a789cf-1dde-4ad9-a877-4facd7d8ab8b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:56.743503 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bca1158-514e-4830-8126-c1680c6f8167 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:56.748862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-4bca1158-514e-4830-8126-c1680c6f8167 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:56.748862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:56.748862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:56.748862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:56.748862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:56.748862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:56.748862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:56.748862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:56.748862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:56.748862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:56.748862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:56.748862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:56.748862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:56.748862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:56.748862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:56.748862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:56.750953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:56.750953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:56.750953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:56.750953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:56.750953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:56.750953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:56.750953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:56.750953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:56.750953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:56.750953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:56.750953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:56.750953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:56.750953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:56.750953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:56.750953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:56.750953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:56.752860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:56.752860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:56.752860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:56.752860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:56.752860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:56.752860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:56.752860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:56.752860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:56.752860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:56.752860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:56.752860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:12:56.752860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:56.752860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 184/371] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:12:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:56.752860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bca1158-514e-4830-8126-c1680c6f8167 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:56.789426 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:56.806306 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2781acd2-786c-4000-b2ef-38abf77ce9b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:56.806918 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2781acd2-786c-4000-b2ef-38abf77ce9b9 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:56.807189 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2781acd2-786c-4000-b2ef-38abf77ce9b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:56.807456 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2781acd2-786c-4000-b2ef-38abf77ce9b9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:56.808186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2781acd2-786c-4000-b2ef-38abf77ce9b9 None admin] RBAC: Adding query filter params (role.id=902fd8d9a4ef48058e1b17e0429ede58, user.id=e8d54ce3bc0746c2a48bbda5024200b7, scope.system=all) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:56.809111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2781acd2-786c-4000-b2ef-38abf77ce9b9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:56.810764 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2781acd2-786c-4000-b2ef-38abf77ce9b9 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:56.837307 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 188/372] 10.222.0.85 () {56 vars in 1308 bytes} [Tue Aug 30 04:12:56 2022] GET /identity/v3/role_assignments?user.id=e8d54ce3bc0746c2a48bbda5024200b7&scope.system=all&role.id=902fd8d9a4ef48058e1b17e0429ede58 => generated 510 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:56.839207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2781acd2-786c-4000-b2ef-38abf77ce9b9 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:56.878890 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:56.919945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bca1158-514e-4830-8126-c1680c6f8167 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:56.969515 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53494562-e267-4aa3-8b10-2d7e9b4fd991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:56.970162 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53494562-e267-4aa3-8b10-2d7e9b4fd991 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:56.970446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53494562-e267-4aa3-8b10-2d7e9b4fd991 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:56.970792 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53494562-e267-4aa3-8b10-2d7e9b4fd991 None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:56.972943 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53494562-e267-4aa3-8b10-2d7e9b4fd991 None admin] RBAC: Authorizing `identity:get_project(project_id=008c6a0a4a114009bc7c692aabd0b0c8)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:56.974509 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53494562-e267-4aa3-8b10-2d7e9b4fd991 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:56.976210 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 185/373] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:56 2022] GET /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8 => generated 302 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:56.983308 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53494562-e267-4aa3-8b10-2d7e9b4fd991 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:57.086880 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2781acd2-786c-4000-b2ef-38abf77ce9b9 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:57.160431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb93f0d7-42f3-439a-bd7b-8153bba6345f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:57.161134 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb93f0d7-42f3-439a-bd7b-8153bba6345f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:57.161356 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb93f0d7-42f3-439a-bd7b-8153bba6345f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:57.162211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb93f0d7-42f3-439a-bd7b-8153bba6345f None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:57.164785 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb93f0d7-42f3-439a-bd7b-8153bba6345f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:57.165119 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb93f0d7-42f3-439a-bd7b-8153bba6345f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:57.166520 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb93f0d7-42f3-439a-bd7b-8153bba6345f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:57.189227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 189/374] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 351 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:57.193552 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb93f0d7-42f3-439a-bd7b-8153bba6345f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:57.211943 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53494562-e267-4aa3-8b10-2d7e9b4fd991 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:57.281505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:57.288752 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf85c32-5e6b-4226-8307-cac6e0084f77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:57.289465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf85c32-5e6b-4226-8307-cac6e0084f77 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:57.291444 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf85c32-5e6b-4226-8307-cac6e0084f77 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:57.291444 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf85c32-5e6b-4226-8307-cac6e0084f77 None admin] PATH_INFO: `/v3/groups/329d8769e54e48bc8acd9188320c782a` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:57.292131 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf85c32-5e6b-4226-8307-cac6e0084f77 None admin] RBAC: Authorizing `identity:get_group(group_id=329d8769e54e48bc8acd9188320c782a)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:57.293831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf85c32-5e6b-4226-8307-cac6e0084f77 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:57.295572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 186/375] 10.222.0.85 () {56 vars in 1180 bytes} [Tue Aug 30 04:12:56 2022] GET /identity/v3/groups/329d8769e54e48bc8acd9188320c782a => generated 226 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:57.304557 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf85c32-5e6b-4226-8307-cac6e0084f77 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:57.307569 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:57.368550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb93f0d7-42f3-439a-bd7b-8153bba6345f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:57.405708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df875e3f-31fb-4de0-b82a-2095a28dfe49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:57.406681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df875e3f-31fb-4de0-b82a-2095a28dfe49 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:57.407173 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df875e3f-31fb-4de0-b82a-2095a28dfe49 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:57.407635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df875e3f-31fb-4de0-b82a-2095a28dfe49 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:57.486864 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df875e3f-31fb-4de0-b82a-2095a28dfe49 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:57.488747 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df875e3f-31fb-4de0-b82a-2095a28dfe49 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:57.491267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf85c32-5e6b-4226-8307-cac6e0084f77 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:57.546876 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:57.551881 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8346207c-19e4-4604-a36d-05b30ccda73b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:57.552815 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8346207c-19e4-4604-a36d-05b30ccda73b None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:57.553332 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8346207c-19e4-4604-a36d-05b30ccda73b None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:57.553849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8346207c-19e4-4604-a36d-05b30ccda73b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:57.554791 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8346207c-19e4-4604-a36d-05b30ccda73b None admin] RBAC: Adding query filter params (group.id=329d8769e54e48bc8acd9188320c782a, role.id=a9e862ac528245ca8363bccc5bf35287, scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:57.555306 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8346207c-19e4-4604-a36d-05b30ccda73b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:57.556946 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8346207c-19e4-4604-a36d-05b30ccda73b None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:57.564869 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 187/376] 10.222.0.85 () {56 vars in 1376 bytes} [Tue Aug 30 04:12:57 2022] GET /identity/v3/role_assignments?group.id=329d8769e54e48bc8acd9188320c782a&scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8&role.id=a9e862ac528245ca8363bccc5bf35287 => generated 260 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:57.569343 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d214ec9-86f8-4afd-9ffd-1834d006caa1 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:57.570835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d214ec9-86f8-4afd-9ffd-1834d006caa1 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:57.572294 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d214ec9-86f8-4afd-9ffd-1834d006caa1 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:57.575669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 188/377] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:57 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:57.582538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5312fa31-4a25-4417-86df-b9cba7747791 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:57.585566 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5312fa31-4a25-4417-86df-b9cba7747791 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:57.585852 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5312fa31-4a25-4417-86df-b9cba7747791 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:57.619099 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-5312fa31-4a25-4417-86df-b9cba7747791 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:57.621728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-5312fa31-4a25-4417-86df-b9cba7747791 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:57.647950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 190/378] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:12:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 455 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:12:57.654439 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df875e3f-31fb-4de0-b82a-2095a28dfe49 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:57.760539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:57.775008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 189/379] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:57.821406 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df875e3f-31fb-4de0-b82a-2095a28dfe49 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:57.860210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2760509a-350e-4158-8636-504a2fb57dd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:57.860845 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2760509a-350e-4158-8636-504a2fb57dd0 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:57.861146 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2760509a-350e-4158-8636-504a2fb57dd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:57.861416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2760509a-350e-4158-8636-504a2fb57dd0 None admin] PATH_INFO: `/v3/projects/be2e356bbe7d4680bd52a62e89c77545` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:57.862540 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2760509a-350e-4158-8636-504a2fb57dd0 None admin] RBAC: Authorizing `identity:get_project(project_id=be2e356bbe7d4680bd52a62e89c77545)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:57.864140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2760509a-350e-4158-8636-504a2fb57dd0 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:57.866162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 191/380] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:57 2022] GET /identity/v3/projects/be2e356bbe7d4680bd52a62e89c77545 => generated 344 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:57.871677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5312fa31-4a25-4417-86df-b9cba7747791 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:57.979783 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5312fa31-4a25-4417-86df-b9cba7747791 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:58.020817 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1ae90d-3853-4478-91c8-4ec72026e0e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:58.021803 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1ae90d-3853-4478-91c8-4ec72026e0e4 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:58.022252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1ae90d-3853-4478-91c8-4ec72026e0e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:58.022715 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1ae90d-3853-4478-91c8-4ec72026e0e4 None admin] PATH_INFO: `/v3/groups/7f39b97c2dba444da6af6f3b248d96ad` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:58.029246 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1ae90d-3853-4478-91c8-4ec72026e0e4 None admin] RBAC: Authorizing `identity:get_group(group_id=7f39b97c2dba444da6af6f3b248d96ad)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:58.030888 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1ae90d-3853-4478-91c8-4ec72026e0e4 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:58.032989 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 190/381] 10.222.0.85 () {56 vars in 1180 bytes} [Tue Aug 30 04:12:57 2022] GET /identity/v3/groups/7f39b97c2dba444da6af6f3b248d96ad => generated 229 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:58.039195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2760509a-350e-4158-8636-504a2fb57dd0 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:58.052148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22ddaa7-865b-4c35-9d7e-2a4122d0e0f3 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:58.052498 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22ddaa7-865b-4c35-9d7e-2a4122d0e0f3 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:58.052778 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22ddaa7-865b-4c35-9d7e-2a4122d0e0f3 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:58.054407 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 191/382] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:58.060247 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fb12e0-209d-4592-9086-0181d8a72f6b None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:58.060526 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fb12e0-209d-4592-9086-0181d8a72f6b None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:58.060798 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fb12e0-209d-4592-9086-0181d8a72f6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:58.063051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:58.098137 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-a8fb12e0-209d-4592-9086-0181d8a72f6b None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:58.099866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-a8fb12e0-209d-4592-9086-0181d8a72f6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:58.160665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2760509a-350e-4158-8636-504a2fb57dd0 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:58.185419 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:58.199251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa0197c-501e-47e0-9f25-f682a66e3607 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:58.199923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 192/383] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:12:58.200197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa0197c-501e-47e0-9f25-f682a66e3607 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:58.200541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa0197c-501e-47e0-9f25-f682a66e3607 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:58.200925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa0197c-501e-47e0-9f25-f682a66e3607 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:58.201449 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fb12e0-209d-4592-9086-0181d8a72f6b None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:58.201833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa0197c-501e-47e0-9f25-f682a66e3607 None admin] RBAC: Adding query filter params (group.id=7f39b97c2dba444da6af6f3b248d96ad, role.id=d17954e7ce544f0d90a512340232c60e, scope.project.id=be2e356bbe7d4680bd52a62e89c77545) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:58.202203 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa0197c-501e-47e0-9f25-f682a66e3607 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:58.209061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa0197c-501e-47e0-9f25-f682a66e3607 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:58.209923 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 192/384] 10.222.0.85 () {56 vars in 1376 bytes} [Tue Aug 30 04:12:58 2022] GET /identity/v3/role_assignments?group.id=7f39b97c2dba444da6af6f3b248d96ad&scope.project.id=be2e356bbe7d4680bd52a62e89c77545&role.id=d17954e7ce544f0d90a512340232c60e => generated 260 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:58.362335 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fb12e0-209d-4592-9086-0181d8a72f6b None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:58.412344 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3566dace-38e9-41bf-957b-c6b6bc83da46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:58.412919 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3566dace-38e9-41bf-957b-c6b6bc83da46 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:58.413229 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3566dace-38e9-41bf-957b-c6b6bc83da46 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:58.413504 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3566dace-38e9-41bf-957b-c6b6bc83da46 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:58.427707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3566dace-38e9-41bf-957b-c6b6bc83da46 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:58.433006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-3566dace-38e9-41bf-957b-c6b6bc83da46 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:12:58.433006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:58.433006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:58.433006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:58.433006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:58.433006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:58.433006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:58.433006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:58.433006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:58.433006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:58.433006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:58.433006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:58.433006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:58.433006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:58.433006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:58.433006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:58.433936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:58.433936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:58.433936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:58.433936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:58.433936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:58.433936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:58.433936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:58.433936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:58.433936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:58.433936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:58.433936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:58.433936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:58.433936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:58.433936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:58.433936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:58.433936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:58.434725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:58.434725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:58.434725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:58.434725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:58.434725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:58.434725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:58.434725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:58.434725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:58.434725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:58.434725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:58.434725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:12:58.434725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:58.434725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 193/385] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:12:58 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:12:58.441810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa0197c-501e-47e0-9f25-f682a66e3607 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:58.549242 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3566dace-38e9-41bf-957b-c6b6bc83da46 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:58.551675 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:58.588140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa0197c-501e-47e0-9f25-f682a66e3607 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:58.649117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c112c7b-d726-4d01-929e-12664549e59c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:58.649741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c112c7b-d726-4d01-929e-12664549e59c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:58.650022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c112c7b-d726-4d01-929e-12664549e59c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:58.650299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c112c7b-d726-4d01-929e-12664549e59c None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:58.650981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c112c7b-d726-4d01-929e-12664549e59c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:58.651285 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c112c7b-d726-4d01-929e-12664549e59c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:58.652673 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c112c7b-d726-4d01-929e-12664549e59c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:58.655547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:58.665030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 193/386] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:12:58 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:58.671683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c112c7b-d726-4d01-929e-12664549e59c None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:58.742994 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3566dace-38e9-41bf-957b-c6b6bc83da46 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:58.801513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c48349-2f6d-488b-b085-1004f0d296ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:58.802259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c48349-2f6d-488b-b085-1004f0d296ef None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:58.802983 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c48349-2f6d-488b-b085-1004f0d296ef None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:58.803253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c48349-2f6d-488b-b085-1004f0d296ef None admin] PATH_INFO: `/v3/roles/member` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:58.821481 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c48349-2f6d-488b-b085-1004f0d296ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:58.826381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-f3c48349-2f6d-488b-b085-1004f0d296ef None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:58.826381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:58.826381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:58.826381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:58.826381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:58.826381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:58.826381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:58.826381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:58.826381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:58.826381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:58.826381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:58.826381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:58.826381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:58.826381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:58.826381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:58.826381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:58.827245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:58.827245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:58.827245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:58.827245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:58.827245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:58.827245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:58.827245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:58.827245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:58.827245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:58.827245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:58.827245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:58.827245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:58.827245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:58.827245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:58.827245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:58.827245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:58.828051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:58.828051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:58.828051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:58.828051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:58.828051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:58.828051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:58.828051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:58.828051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:58.828051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:58.828051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:58.828051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:58.828051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:58.828635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 194/387] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:58 2022] GET /identity/v3/roles/member => generated 84 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:12:58.837152 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c112c7b-d726-4d01-929e-12664549e59c None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:58.838882 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c48349-2f6d-488b-b085-1004f0d296ef None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:58.893733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc76c33-94d1-46c7-b743-0ecd124309e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:58.894816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc76c33-94d1-46c7-b743-0ecd124309e3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:58.895309 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc76c33-94d1-46c7-b743-0ecd124309e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:58.895775 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc76c33-94d1-46c7-b743-0ecd124309e3 None admin] PATH_INFO: `/v3/groups/329d8769e54e48bc8acd9188320c782a` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:58.898506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc76c33-94d1-46c7-b743-0ecd124309e3 None admin] RBAC: Authorizing `identity:get_group(group_id=329d8769e54e48bc8acd9188320c782a)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:58.900299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc76c33-94d1-46c7-b743-0ecd124309e3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:58.903479 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 194/388] 10.222.0.85 () {56 vars in 1180 bytes} [Tue Aug 30 04:12:58 2022] GET /identity/v3/groups/329d8769e54e48bc8acd9188320c782a => generated 226 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:58.911022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc76c33-94d1-46c7-b743-0ecd124309e3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:58.935226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:58.993012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c48349-2f6d-488b-b085-1004f0d296ef None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:59.004644 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:59.037878 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40555610-33b5-4efa-8e38-fca426bc11e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:59.038539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40555610-33b5-4efa-8e38-fca426bc11e6 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:59.038821 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40555610-33b5-4efa-8e38-fca426bc11e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:59.039091 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40555610-33b5-4efa-8e38-fca426bc11e6 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:59.039764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40555610-33b5-4efa-8e38-fca426bc11e6 None admin] RBAC: Adding query filter params (name=member) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:59.040062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40555610-33b5-4efa-8e38-fca426bc11e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:59.042814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40555610-33b5-4efa-8e38-fca426bc11e6 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:59.055583 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 195/389] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:58 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:59.056950 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680bf15-1560-4b6b-96e7-a1878b7e6ff9 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:59.057256 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680bf15-1560-4b6b-96e7-a1878b7e6ff9 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:59.057527 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680bf15-1560-4b6b-96e7-a1878b7e6ff9 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:59.057820 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc76c33-94d1-46c7-b743-0ecd124309e3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:59.061109 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 196/390] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:12:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:12:59.062078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9680bf15-1560-4b6b-96e7-a1878b7e6ff9 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:59.098500 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e39eb1-7b03-4003-b69f-679d315041bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:59.099737 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e39eb1-7b03-4003-b69f-679d315041bb None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:59.099995 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e39eb1-7b03-4003-b69f-679d315041bb None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:59.100271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e39eb1-7b03-4003-b69f-679d315041bb None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:59.101474 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e39eb1-7b03-4003-b69f-679d315041bb None admin] RBAC: Authorizing `identity:get_project(project_id=26661cf5060b4b939d8ab02f9ea45daf)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:59.102980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e39eb1-7b03-4003-b69f-679d315041bb None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:59.109130 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 195/391] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:58 2022] GET /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf => generated 298 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:59.109130 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905b6ac7-2815-464c-a4c0-d508bfa54117 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:59.109130 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905b6ac7-2815-464c-a4c0-d508bfa54117 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:59.109130 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905b6ac7-2815-464c-a4c0-d508bfa54117 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:59.126031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-905b6ac7-2815-464c-a4c0-d508bfa54117 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:12:59.127242 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-905b6ac7-2815-464c-a4c0-d508bfa54117 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:12:59.205196 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 196/392] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:12:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:12:59.206873 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905b6ac7-2815-464c-a4c0-d508bfa54117 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:59.211549 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9680bf15-1560-4b6b-96e7-a1878b7e6ff9 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:59.225514 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:59.304981 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723ebdbd-39c1-4990-9e43-2d58ae8fa023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:59.309993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723ebdbd-39c1-4990-9e43-2d58ae8fa023 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:59.310477 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723ebdbd-39c1-4990-9e43-2d58ae8fa023 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:59.310944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723ebdbd-39c1-4990-9e43-2d58ae8fa023 None admin] PATH_INFO: `/v3/groups/329d8769e54e48bc8acd9188320c782a` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:59.312278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723ebdbd-39c1-4990-9e43-2d58ae8fa023 None admin] RBAC: Authorizing `identity:get_group(group_id=329d8769e54e48bc8acd9188320c782a)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:59.313919 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723ebdbd-39c1-4990-9e43-2d58ae8fa023 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:59.321171 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 197/393] 10.222.0.85 () {56 vars in 1180 bytes} [Tue Aug 30 04:12:59 2022] GET /identity/v3/groups/329d8769e54e48bc8acd9188320c782a => generated 226 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:12:59.325950 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723ebdbd-39c1-4990-9e43-2d58ae8fa023 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:59.361463 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905b6ac7-2815-464c-a4c0-d508bfa54117 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:59.395725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:59.458424 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e562d59-6537-4243-a4d5-30e4b2d2f03d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:59.459122 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e562d59-6537-4243-a4d5-30e4b2d2f03d None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:59.459414 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e562d59-6537-4243-a4d5-30e4b2d2f03d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:59.459695 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e562d59-6537-4243-a4d5-30e4b2d2f03d None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf/groups/329d8769e54e48bc8acd9188320c782a/roles/a9e862ac528245ca8363bccc5bf35287` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:59.466796 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723ebdbd-39c1-4990-9e43-2d58ae8fa023 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:59.469799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e562d59-6537-4243-a4d5-30e4b2d2f03d None admin] RBAC: Authorizing `identity:create_grant(project_id=26661cf5060b4b939d8ab02f9ea45daf, group_id=329d8769e54e48bc8acd9188320c782a, role_id=a9e862ac528245ca8363bccc5bf35287)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:59.471452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e562d59-6537-4243-a4d5-30e4b2d2f03d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:59.503468 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 197/394] 10.222.0.85 () {58 vars in 1442 bytes} [Tue Aug 30 04:12:59 2022] PUT /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf/groups/329d8769e54e48bc8acd9188320c782a/roles/a9e862ac528245ca8363bccc5bf35287 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:12:59.509973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7da9f1-4752-4c14-b18e-643b48400229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:59.510701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7da9f1-4752-4c14-b18e-643b48400229 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:59.511042 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7da9f1-4752-4c14-b18e-643b48400229 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:59.511381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7da9f1-4752-4c14-b18e-643b48400229 None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:59.512677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7da9f1-4752-4c14-b18e-643b48400229 None admin] RBAC: Authorizing `identity:get_project(project_id=26661cf5060b4b939d8ab02f9ea45daf)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:59.514221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7da9f1-4752-4c14-b18e-643b48400229 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:59.516221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 198/395] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:12:59 2022] GET /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf => generated 298 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:59.522166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e562d59-6537-4243-a4d5-30e4b2d2f03d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:59.561360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7da9f1-4752-4c14-b18e-643b48400229 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:59.688171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:59.697176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7da9f1-4752-4c14-b18e-643b48400229 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:59.708730 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:12:59.739792 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003f5310-92da-45fc-88fa-7f3e7acab41a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:59.740478 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003f5310-92da-45fc-88fa-7f3e7acab41a None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:59.740752 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003f5310-92da-45fc-88fa-7f3e7acab41a None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:59.741025 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003f5310-92da-45fc-88fa-7f3e7acab41a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:59.753342 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003f5310-92da-45fc-88fa-7f3e7acab41a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:12:59.760213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-003f5310-92da-45fc-88fa-7f3e7acab41a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:59.760213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:12:59.760213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:12:59.760213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:12:59.760213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:12:59.760213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:12:59.760213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:12:59.760213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:12:59.760213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:12:59.760213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:12:59.760213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:12:59.760213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:12:59.760213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:12:59.760213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:12:59.760213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:12:59.760213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:12:59.761282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:12:59.761282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:12:59.761282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:12:59.761282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:12:59.761282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:12:59.761282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:12:59.761282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:12:59.761282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:12:59.761282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:12:59.761282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:12:59.761282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:12:59.761282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:12:59.761282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:12:59.761282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:12:59.761282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:12:59.761282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:12:59.762140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:12:59.762140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:12:59.762140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:12:59.762140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:12:59.762140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:12:59.762140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:12:59.762140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:12:59.762140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:12:59.762140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:12:59.762140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:12:59.762140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:12:59.762140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:12:59.763904 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 199/396] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:12:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:12:59.766821 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e562d59-6537-4243-a4d5-30e4b2d2f03d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:59.770326 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003f5310-92da-45fc-88fa-7f3e7acab41a None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:12:59.815134 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35631224-0838-497b-8793-02935e92e4da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:59.816231 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35631224-0838-497b-8793-02935e92e4da None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:59.816687 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35631224-0838-497b-8793-02935e92e4da None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:59.817296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35631224-0838-497b-8793-02935e92e4da None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf/groups/329d8769e54e48bc8acd9188320c782a/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:59.821531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35631224-0838-497b-8793-02935e92e4da None admin] RBAC: Authorizing `identity:create_grant(project_id=26661cf5060b4b939d8ab02f9ea45daf, group_id=329d8769e54e48bc8acd9188320c782a, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:59.824440 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35631224-0838-497b-8793-02935e92e4da None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:59.844737 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 198/397] 10.222.0.85 () {58 vars in 1442 bytes} [Tue Aug 30 04:12:59 2022] PUT /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf/groups/329d8769e54e48bc8acd9188320c782a/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:12:59.897183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003f5310-92da-45fc-88fa-7f3e7acab41a None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:12:59.935979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99315d22-2712-430c-b298-d2b4b8bc8e60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:12:59.936605 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99315d22-2712-430c-b298-d2b4b8bc8e60 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:12:59.936882 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99315d22-2712-430c-b298-d2b4b8bc8e60 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:12:59.937234 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99315d22-2712-430c-b298-d2b4b8bc8e60 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:12:59.937924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99315d22-2712-430c-b298-d2b4b8bc8e60 None admin] RBAC: Adding query filter params (name=member) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:12:59.938223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99315d22-2712-430c-b298-d2b4b8bc8e60 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:12:59.939659 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99315d22-2712-430c-b298-d2b4b8bc8e60 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:12:59.950069 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 200/398] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:12:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:12:59.955754 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35631224-0838-497b-8793-02935e92e4da None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:00.088366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35631224-0838-497b-8793-02935e92e4da None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:00.118974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:00.132298 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4086c345-204d-423f-afd2-098b9e5fabbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:00.132960 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4086c345-204d-423f-afd2-098b9e5fabbf None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:00.133276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4086c345-204d-423f-afd2-098b9e5fabbf None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:00.133806 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4086c345-204d-423f-afd2-098b9e5fabbf None admin] PATH_INFO: `/v3/groups/329d8769e54e48bc8acd9188320c782a` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:00.135031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4086c345-204d-423f-afd2-098b9e5fabbf None admin] RBAC: Authorizing `identity:get_group(group_id=329d8769e54e48bc8acd9188320c782a)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:00.136843 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4086c345-204d-423f-afd2-098b9e5fabbf None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:00.139007 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 199/399] 10.222.0.85 () {56 vars in 1180 bytes} [Tue Aug 30 04:12:59 2022] GET /identity/v3/groups/329d8769e54e48bc8acd9188320c782a => generated 226 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:00.144760 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99315d22-2712-430c-b298-d2b4b8bc8e60 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:00.182694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:00.254423 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1c8f14-35c2-46ce-a636-17c52af7003f None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:00.254925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1c8f14-35c2-46ce-a636-17c52af7003f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:00.254999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1c8f14-35c2-46ce-a636-17c52af7003f None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:00.257123 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 200/400] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:13:00.262532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eb575a-de2d-4263-bb61-4d7e1a9c4d47 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:00.262813 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eb575a-de2d-4263-bb61-4d7e1a9c4d47 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:00.263091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eb575a-de2d-4263-bb61-4d7e1a9c4d47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:00.274830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99315d22-2712-430c-b298-d2b4b8bc8e60 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:00.297110 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b6eb575a-de2d-4263-bb61-4d7e1a9c4d47 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:00.298381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b6eb575a-de2d-4263-bb61-4d7e1a9c4d47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:00.323573 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5380c2b-f0bf-4a1e-b8e8-28d5bbb18e16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:00.324338 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5380c2b-f0bf-4a1e-b8e8-28d5bbb18e16 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:00.324623 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5380c2b-f0bf-4a1e-b8e8-28d5bbb18e16 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:00.324893 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5380c2b-f0bf-4a1e-b8e8-28d5bbb18e16 None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:00.327188 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5380c2b-f0bf-4a1e-b8e8-28d5bbb18e16 None admin] RBAC: Authorizing `identity:get_project(project_id=008c6a0a4a114009bc7c692aabd0b0c8)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:00.328793 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5380c2b-f0bf-4a1e-b8e8-28d5bbb18e16 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:00.331914 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 201/401] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:13:00 2022] GET /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8 => generated 302 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:00.338321 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5380c2b-f0bf-4a1e-b8e8-28d5bbb18e16 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:00.412679 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 201/402] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:13:00.469767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:00.474074 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5380c2b-f0bf-4a1e-b8e8-28d5bbb18e16 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:00.519956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd32943c-40aa-4d49-a705-ab4c3e60f80c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:00.520634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd32943c-40aa-4d49-a705-ab4c3e60f80c None admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:00.520964 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd32943c-40aa-4d49-a705-ab4c3e60f80c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:00.521312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd32943c-40aa-4d49-a705-ab4c3e60f80c None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8/groups/329d8769e54e48bc8acd9188320c782a/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:00.523273 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd32943c-40aa-4d49-a705-ab4c3e60f80c None admin] RBAC: Authorizing `identity:create_grant(project_id=008c6a0a4a114009bc7c692aabd0b0c8, group_id=329d8769e54e48bc8acd9188320c782a, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:00.524710 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd32943c-40aa-4d49-a705-ab4c3e60f80c None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:00.543321 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 202/403] 10.222.0.85 () {58 vars in 1442 bytes} [Tue Aug 30 04:13:00 2022] PUT /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8/groups/329d8769e54e48bc8acd9188320c782a/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:13:00.806538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6eb575a-de2d-4263-bb61-4d7e1a9c4d47 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:00.823012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29a7c91-6e7b-4556-8a76-1b54fcb68676 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:00.823355 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29a7c91-6e7b-4556-8a76-1b54fcb68676 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:00.823565 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29a7c91-6e7b-4556-8a76-1b54fcb68676 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:00.825401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 203/404] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:13:00.829900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79737da-cd73-4721-9633-90639fa82cda None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:00.830182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79737da-cd73-4721-9633-90639fa82cda None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:00.830451 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79737da-cd73-4721-9633-90639fa82cda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:00.861459 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-c79737da-cd73-4721-9633-90639fa82cda None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:00.862655 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c79737da-cd73-4721-9633-90639fa82cda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:00.983145 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 204/405] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:01.000993 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6eb575a-de2d-4263-bb61-4d7e1a9c4d47 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:01.048123 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:01.063605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51964e7-b7fe-47df-a294-5adb23b5b7bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:01.064162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51964e7-b7fe-47df-a294-5adb23b5b7bd None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:01.064434 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51964e7-b7fe-47df-a294-5adb23b5b7bd None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:01.064710 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51964e7-b7fe-47df-a294-5adb23b5b7bd None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:01.085959 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51964e7-b7fe-47df-a294-5adb23b5b7bd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:01.090782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-e51964e7-b7fe-47df-a294-5adb23b5b7bd None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:13:01.090782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:01.090782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:01.090782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:01.090782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:01.090782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:01.090782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:01.090782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:01.090782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:01.090782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:01.090782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:01.090782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:01.090782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:01.090782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:01.090782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:01.090782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:01.091655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:01.091655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:01.091655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:01.091655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:01.091655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:01.091655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:01.091655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:01.091655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:01.091655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:01.091655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:01.091655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:01.091655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:01.091655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:01.091655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:01.091655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:01.091655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:01.092458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:01.092458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:01.092458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:01.092458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:01.092458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:01.092458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:01.092458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:01.092458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:01.092458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:01.092458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:01.092458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:13:01.092458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:01.092458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 202/406] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:13:00 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:01.099794 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79737da-cd73-4721-9633-90639fa82cda None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:01.238651 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:01.252169 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79737da-cd73-4721-9633-90639fa82cda None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:01.300811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3b063f-96d2-421e-96d5-cd495d13fbbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:01.301452 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3b063f-96d2-421e-96d5-cd495d13fbbe None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:01.301730 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3b063f-96d2-421e-96d5-cd495d13fbbe None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:01.301997 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3b063f-96d2-421e-96d5-cd495d13fbbe None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:01.302696 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3b063f-96d2-421e-96d5-cd495d13fbbe None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:01.302995 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3b063f-96d2-421e-96d5-cd495d13fbbe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:01.304372 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3b063f-96d2-421e-96d5-cd495d13fbbe None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:01.310352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51964e7-b7fe-47df-a294-5adb23b5b7bd None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:01.318614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 205/407] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:01.324522 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3b063f-96d2-421e-96d5-cd495d13fbbe None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:01.458640 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3b063f-96d2-421e-96d5-cd495d13fbbe None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:01.502699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51964e7-b7fe-47df-a294-5adb23b5b7bd None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:01.533720 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f5fdd8-5455-47cd-8af9-0160e30230b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:01.534349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f5fdd8-5455-47cd-8af9-0160e30230b9 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:01.534721 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f5fdd8-5455-47cd-8af9-0160e30230b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:01.534993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f5fdd8-5455-47cd-8af9-0160e30230b9 None admin] PATH_INFO: `/v3/groups/329d8769e54e48bc8acd9188320c782a` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:01.541622 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22a4808-7af3-4b70-a242-a9fd2848220e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:01.541879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f5fdd8-5455-47cd-8af9-0160e30230b9 None admin] RBAC: Authorizing `identity:get_group(group_id=329d8769e54e48bc8acd9188320c782a)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:01.542695 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22a4808-7af3-4b70-a242-a9fd2848220e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:01.543170 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22a4808-7af3-4b70-a242-a9fd2848220e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:01.543273 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f5fdd8-5455-47cd-8af9-0160e30230b9 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:01.543742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22a4808-7af3-4b70-a242-a9fd2848220e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:01.545215 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 206/408] 10.222.0.85 () {56 vars in 1180 bytes} [Tue Aug 30 04:13:01 2022] GET /identity/v3/groups/329d8769e54e48bc8acd9188320c782a => generated 226 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:01.551142 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f5fdd8-5455-47cd-8af9-0160e30230b9 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:01.557691 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22a4808-7af3-4b70-a242-a9fd2848220e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:01.563050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-f22a4808-7af3-4b70-a242-a9fd2848220e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:13:01.563050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:01.563050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:01.563050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:01.563050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:01.563050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:01.563050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:01.563050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:01.563050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:01.563050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:01.563050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:01.563050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:01.563050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:01.563050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:01.563050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:01.563050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:01.563999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:01.563999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:01.563999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:01.563999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:01.563999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:01.563999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:01.563999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:01.563999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:01.563999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:01.563999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:01.563999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:01.563999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:01.563999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:01.563999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:01.563999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:01.563999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:01.564783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:01.564783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:01.564783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:01.564783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:01.564783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:01.564783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:01.564783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:01.564783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:01.564783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:01.564783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:01.564783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:13:01.564783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:01.570495 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 203/409] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:13:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:01.579786 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22a4808-7af3-4b70-a242-a9fd2848220e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:01.588664 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:01.658683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:01.709737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f5fdd8-5455-47cd-8af9-0160e30230b9 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:01.716890 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22a4808-7af3-4b70-a242-a9fd2848220e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:01.747764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e814e04-3b11-406f-a196-3d173ff7a098 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:01.748647 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e814e04-3b11-406f-a196-3d173ff7a098 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:01.749367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e814e04-3b11-406f-a196-3d173ff7a098 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:01.749654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e814e04-3b11-406f-a196-3d173ff7a098 None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:01.750785 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e814e04-3b11-406f-a196-3d173ff7a098 None admin] RBAC: Authorizing `identity:get_project(project_id=008c6a0a4a114009bc7c692aabd0b0c8)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:01.752221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e814e04-3b11-406f-a196-3d173ff7a098 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:01.754563 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 207/410] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:13:01 2022] GET /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8 => generated 302 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:01.759969 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e814e04-3b11-406f-a196-3d173ff7a098 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:01.766078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6825d310-90d4-4eb5-96a9-ec6f5db28139 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:01.766701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6825d310-90d4-4eb5-96a9-ec6f5db28139 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:01.766981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6825d310-90d4-4eb5-96a9-ec6f5db28139 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:01.767253 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6825d310-90d4-4eb5-96a9-ec6f5db28139 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:01.767926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6825d310-90d4-4eb5-96a9-ec6f5db28139 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:01.768228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6825d310-90d4-4eb5-96a9-ec6f5db28139 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:01.769626 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6825d310-90d4-4eb5-96a9-ec6f5db28139 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:01.790897 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 204/411] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:13:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:01.794522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6825d310-90d4-4eb5-96a9-ec6f5db28139 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:01.952871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6825d310-90d4-4eb5-96a9-ec6f5db28139 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:01.959385 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:01.968081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:01.980793 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e814e04-3b11-406f-a196-3d173ff7a098 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:01.994282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29426760-21bf-48f6-83ac-03fc4fe094eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:01.994926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29426760-21bf-48f6-83ac-03fc4fe094eb None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:01.995205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29426760-21bf-48f6-83ac-03fc4fe094eb None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:01.995486 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29426760-21bf-48f6-83ac-03fc4fe094eb None admin] PATH_INFO: `/v3/groups/7f39b97c2dba444da6af6f3b248d96ad` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:01.996590 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29426760-21bf-48f6-83ac-03fc4fe094eb None admin] RBAC: Authorizing `identity:get_group(group_id=7f39b97c2dba444da6af6f3b248d96ad)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:01.998063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29426760-21bf-48f6-83ac-03fc4fe094eb None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:01.999983 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 205/412] 10.222.0.85 () {56 vars in 1180 bytes} [Tue Aug 30 04:13:01 2022] GET /identity/v3/groups/7f39b97c2dba444da6af6f3b248d96ad => generated 229 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:02.005723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29426760-21bf-48f6-83ac-03fc4fe094eb None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:02.054459 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43c5dd6-7153-4399-b109-03d7c69ef607 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:02.055228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43c5dd6-7153-4399-b109-03d7c69ef607 None admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:02.055510 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43c5dd6-7153-4399-b109-03d7c69ef607 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:02.055779 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43c5dd6-7153-4399-b109-03d7c69ef607 None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8/groups/329d8769e54e48bc8acd9188320c782a/roles/a9e862ac528245ca8363bccc5bf35287` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:02.060570 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43c5dd6-7153-4399-b109-03d7c69ef607 None admin] RBAC: Authorizing `identity:create_grant(project_id=008c6a0a4a114009bc7c692aabd0b0c8, group_id=329d8769e54e48bc8acd9188320c782a, role_id=a9e862ac528245ca8363bccc5bf35287)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:02.062156 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43c5dd6-7153-4399-b109-03d7c69ef607 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:02.078262 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 208/413] 10.222.0.85 () {58 vars in 1442 bytes} [Tue Aug 30 04:13:01 2022] PUT /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8/groups/329d8769e54e48bc8acd9188320c782a/roles/a9e862ac528245ca8363bccc5bf35287 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:13:02.090182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b57f6d5-a02e-4726-9e3d-2e155c26b7c3 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:02.090812 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b57f6d5-a02e-4726-9e3d-2e155c26b7c3 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:02.091311 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b57f6d5-a02e-4726-9e3d-2e155c26b7c3 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:02.093186 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 209/414] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:13:02.099015 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13c230b-9115-46bf-9c00-e40a972b8c75 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:02.099653 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13c230b-9115-46bf-9c00-e40a972b8c75 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:02.100140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13c230b-9115-46bf-9c00-e40a972b8c75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:02.114531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29426760-21bf-48f6-83ac-03fc4fe094eb None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:02.120101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-b13c230b-9115-46bf-9c00-e40a972b8c75 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:02.121634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b13c230b-9115-46bf-9c00-e40a972b8c75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:02.153385 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10d4b9f-bbe2-45f9-8c4f-4172c46bb97b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:02.154038 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10d4b9f-bbe2-45f9-8c4f-4172c46bb97b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:02.154419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10d4b9f-bbe2-45f9-8c4f-4172c46bb97b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:02.154700 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10d4b9f-bbe2-45f9-8c4f-4172c46bb97b None admin] PATH_INFO: `/v3/projects/be2e356bbe7d4680bd52a62e89c77545` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:02.155882 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10d4b9f-bbe2-45f9-8c4f-4172c46bb97b None admin] RBAC: Authorizing `identity:get_project(project_id=be2e356bbe7d4680bd52a62e89c77545)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:02.157529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10d4b9f-bbe2-45f9-8c4f-4172c46bb97b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:02.159665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 206/415] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:13:02 2022] GET /identity/v3/projects/be2e356bbe7d4680bd52a62e89c77545 => generated 344 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:02.170145 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10d4b9f-bbe2-45f9-8c4f-4172c46bb97b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:02.215374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 210/416] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:13:02.241477 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:02.293524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10d4b9f-bbe2-45f9-8c4f-4172c46bb97b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:02.332296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da04357-304e-43bb-85bb-fc1c6c0f62a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:02.333082 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da04357-304e-43bb-85bb-fc1c6c0f62a7 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:02.333303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da04357-304e-43bb-85bb-fc1c6c0f62a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:02.334153 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da04357-304e-43bb-85bb-fc1c6c0f62a7 None admin] PATH_INFO: `/v3/projects/be2e356bbe7d4680bd52a62e89c77545/groups/7f39b97c2dba444da6af6f3b248d96ad/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:02.336181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da04357-304e-43bb-85bb-fc1c6c0f62a7 None admin] RBAC: Authorizing `identity:create_grant(project_id=be2e356bbe7d4680bd52a62e89c77545, group_id=7f39b97c2dba444da6af6f3b248d96ad, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:02.337677 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da04357-304e-43bb-85bb-fc1c6c0f62a7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:02.354674 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 207/417] 10.222.0.85 () {58 vars in 1442 bytes} [Tue Aug 30 04:13:02 2022] PUT /identity/v3/projects/be2e356bbe7d4680bd52a62e89c77545/groups/7f39b97c2dba444da6af6f3b248d96ad/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:13:02.565030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13c230b-9115-46bf-9c00-e40a972b8c75 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:02.574407 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95fd264-6095-4809-9425-6e9c0800ddbd None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:02.574879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95fd264-6095-4809-9425-6e9c0800ddbd None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:02.575223 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95fd264-6095-4809-9425-6e9c0800ddbd None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:02.576644 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 208/418] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:13:02.581828 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9cfb53-f745-4480-84ae-8573968640bb None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:02.582110 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9cfb53-f745-4480-84ae-8573968640bb None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:02.582388 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9cfb53-f745-4480-84ae-8573968640bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:02.603054 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5b9cfb53-f745-4480-84ae-8573968640bb None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:02.604264 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5b9cfb53-f745-4480-84ae-8573968640bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:02.689396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:02.722899 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 209/419] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:13:02.731955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13c230b-9115-46bf-9c00-e40a972b8c75 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:02.790304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdd4bcb-11f6-4d7f-878a-5408ec7228f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:02.790962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdd4bcb-11f6-4d7f-878a-5408ec7228f1 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:02.791244 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdd4bcb-11f6-4d7f-878a-5408ec7228f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:02.791519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdd4bcb-11f6-4d7f-878a-5408ec7228f1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:02.804724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdd4bcb-11f6-4d7f-878a-5408ec7228f1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:02.809685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-5fdd4bcb-11f6-4d7f-878a-5408ec7228f1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:13:02.809685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:02.809685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:02.809685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:02.809685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:02.809685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:02.809685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:02.809685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:02.809685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:02.809685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:02.809685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:02.809685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:02.809685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:02.809685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:02.809685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:02.809685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:02.810639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:02.810639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:02.810639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:02.810639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:02.810639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:02.810639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:02.810639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:02.810639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:02.810639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:02.810639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:02.810639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:02.810639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:02.810639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:02.810639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:02.810639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:02.810639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:02.811462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:02.811462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:02.811462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:02.811462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:02.811462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:02.811462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:02.811462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:02.811462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:02.811462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:02.811462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:02.811462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:13:02.811462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:02.811462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 211/420] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:13:02 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:02.816803 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9cfb53-f745-4480-84ae-8573968640bb None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:02.885510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:02.891495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f0b546-5932-4a9d-ba7a-a4de83903b4e None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:02.892317 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f0b546-5932-4a9d-ba7a-a4de83903b4e None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:02.892317 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f0b546-5932-4a9d-ba7a-a4de83903b4e None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:02.898361 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 212/421] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:02 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:13:02.903830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80f8f60-a179-4384-a745-31e37c99248c None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:02.904115 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80f8f60-a179-4384-a745-31e37c99248c None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:02.904391 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80f8f60-a179-4384-a745-31e37c99248c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:02.940752 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-e80f8f60-a179-4384-a745-31e37c99248c None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:02.941990 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e80f8f60-a179-4384-a745-31e37c99248c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:02.964757 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9cfb53-f745-4480-84ae-8573968640bb None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:03.029714 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9b4bb9-eaae-4dd8-95c9-fa845dc97346 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:03.030695 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9b4bb9-eaae-4dd8-95c9-fa845dc97346 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:03.031186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9b4bb9-eaae-4dd8-95c9-fa845dc97346 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:03.031655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9b4bb9-eaae-4dd8-95c9-fa845dc97346 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:03.032559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9b4bb9-eaae-4dd8-95c9-fa845dc97346 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:03.033077 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9b4bb9-eaae-4dd8-95c9-fa845dc97346 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:03.034645 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9b4bb9-eaae-4dd8-95c9-fa845dc97346 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:03.046575 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 213/422] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:03.059334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 210/423] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:02 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:03.064890 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80f8f60-a179-4384-a745-31e37c99248c None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:03.105299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9b4bb9-eaae-4dd8-95c9-fa845dc97346 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:03.173639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:03.202782 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80f8f60-a179-4384-a745-31e37c99248c None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:03.238681 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a1f9f2-aab6-4e79-8b18-879aa02da6da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:03.239290 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a1f9f2-aab6-4e79-8b18-879aa02da6da None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:03.239569 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a1f9f2-aab6-4e79-8b18-879aa02da6da None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:03.240356 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a1f9f2-aab6-4e79-8b18-879aa02da6da None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:03.300713 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9b4bb9-eaae-4dd8-95c9-fa845dc97346 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:03.309497 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a1f9f2-aab6-4e79-8b18-879aa02da6da None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:03.311086 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a1f9f2-aab6-4e79-8b18-879aa02da6da None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:03.313356 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:03.377639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:03.386583 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b75dad-97c6-4835-965d-c6e31cfd47e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:03.387229 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b75dad-97c6-4835-965d-c6e31cfd47e9 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:03.387499 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b75dad-97c6-4835-965d-c6e31cfd47e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:03.387771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b75dad-97c6-4835-965d-c6e31cfd47e9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:03.413639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b75dad-97c6-4835-965d-c6e31cfd47e9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:03.414717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 214/424] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:13:03.420332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-00b75dad-97c6-4835-965d-c6e31cfd47e9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:13:03.420332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:03.420332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:03.420332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:03.420332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:03.420332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:03.420332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:03.420332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:03.420332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:03.420332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:03.420332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:03.420332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:03.420332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:03.420332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:03.420332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:03.420332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:03.421265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:03.421265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:03.421265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:03.421265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:03.421265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:03.421265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:03.421265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:03.421265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:03.421265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:03.421265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:03.421265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:03.421265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:03.421265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:03.421265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:03.421265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:03.421265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:03.422051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:03.422051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:03.422051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:03.422051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:03.422051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:03.422051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:03.422051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:03.422051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:03.422051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:03.422051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:03.422051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:13:03.422051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:03.422051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 211/425] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:13:03 2022] GET /identity/v3/roles/member => generated 84 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:03.423608 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a1f9f2-aab6-4e79-8b18-879aa02da6da None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:03.431351 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b75dad-97c6-4835-965d-c6e31cfd47e9 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:03.557302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b75dad-97c6-4835-965d-c6e31cfd47e9 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:03.559306 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a1f9f2-aab6-4e79-8b18-879aa02da6da None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:03.596643 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a475c371-7297-42c0-bd71-d8b45237f10b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:03.597651 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a475c371-7297-42c0-bd71-d8b45237f10b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:03.598085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a475c371-7297-42c0-bd71-d8b45237f10b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:03.598503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a475c371-7297-42c0-bd71-d8b45237f10b None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:03.599388 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a475c371-7297-42c0-bd71-d8b45237f10b None admin] RBAC: Adding query filter params (name=member) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:03.599688 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a475c371-7297-42c0-bd71-d8b45237f10b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:03.600412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3336f515-3fe6-489b-869b-f02acb1e68fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:03.601078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a475c371-7297-42c0-bd71-d8b45237f10b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:03.601811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3336f515-3fe6-489b-869b-f02acb1e68fb None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:03.602264 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3336f515-3fe6-489b-869b-f02acb1e68fb None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:03.602796 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3336f515-3fe6-489b-869b-f02acb1e68fb None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:03.604211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3336f515-3fe6-489b-869b-f02acb1e68fb None admin] RBAC: Authorizing `identity:get_project(project_id=26661cf5060b4b939d8ab02f9ea45daf)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:03.605917 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3336f515-3fe6-489b-869b-f02acb1e68fb None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:03.608143 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 215/426] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:13:03 2022] GET /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf => generated 298 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:03.609180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3336f515-3fe6-489b-869b-f02acb1e68fb None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:03.609883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 212/427] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:13:03 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:03.615366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a475c371-7297-42c0-bd71-d8b45237f10b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:03.636058 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:03.648950 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:03.719809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3336f515-3fe6-489b-869b-f02acb1e68fb None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:03.724064 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a475c371-7297-42c0-bd71-d8b45237f10b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:03.757240 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b282ff-ba16-491e-a1d7-74137cdd898a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:03.758132 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b282ff-ba16-491e-a1d7-74137cdd898a None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:03.758581 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b282ff-ba16-491e-a1d7-74137cdd898a None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:03.759045 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b282ff-ba16-491e-a1d7-74137cdd898a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:03.759372 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6ea7af-8f4b-4c26-ad3d-013e40b9623f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:03.759974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6ea7af-8f4b-4c26-ad3d-013e40b9623f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:03.760255 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6ea7af-8f4b-4c26-ad3d-013e40b9623f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:03.760521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6ea7af-8f4b-4c26-ad3d-013e40b9623f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:03.770688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b282ff-ba16-491e-a1d7-74137cdd898a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:03.776747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-c0b282ff-ba16-491e-a1d7-74137cdd898a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:13:03.776747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:03.776747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:03.776747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:03.776747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:03.776747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:03.776747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:03.776747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:03.776747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:03.776747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:03.776747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:03.776747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:03.776747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:03.776747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:03.776747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:03.776747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:03.777654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:03.777654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:03.777654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:03.777654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:03.777654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:03.777654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:03.777654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:03.777654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:03.777654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:03.777654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:03.777654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:03.777654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:03.777654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:03.777654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:03.777654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:03.777654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:03.778461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:03.778461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:03.778461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:03.778461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:03.778461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:03.778461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:03.778461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:03.778461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:03.778461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:03.778461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:03.778461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 04:13:03.778461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:03.785220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 216/428] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:13:03 2022] GET /identity/v3/roles/member => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:03.785220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b282ff-ba16-491e-a1d7-74137cdd898a None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:03.836967 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6ea7af-8f4b-4c26-ad3d-013e40b9623f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:03.838627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6ea7af-8f4b-4c26-ad3d-013e40b9623f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:03.858776 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:03.892928 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:03.900538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b282ff-ba16-491e-a1d7-74137cdd898a None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:03.937664 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c724bb-71e5-4f95-a8b1-fb3dfffb4050 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:03.938698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c724bb-71e5-4f95-a8b1-fb3dfffb4050 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:03.939207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c724bb-71e5-4f95-a8b1-fb3dfffb4050 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:03.939679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c724bb-71e5-4f95-a8b1-fb3dfffb4050 None admin] PATH_INFO: `/v3/groups/329d8769e54e48bc8acd9188320c782a` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:03.941262 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c724bb-71e5-4f95-a8b1-fb3dfffb4050 None admin] RBAC: Authorizing `identity:get_group(group_id=329d8769e54e48bc8acd9188320c782a)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:03.943090 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c724bb-71e5-4f95-a8b1-fb3dfffb4050 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:03.945289 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 217/429] 10.222.0.85 () {56 vars in 1180 bytes} [Tue Aug 30 04:13:03 2022] GET /identity/v3/groups/329d8769e54e48bc8acd9188320c782a => generated 226 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:03.946276 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c724bb-71e5-4f95-a8b1-fb3dfffb4050 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:03.952305 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 213/430] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:13:03.953238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6ea7af-8f4b-4c26-ad3d-013e40b9623f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:04.055152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c724bb-71e5-4f95-a8b1-fb3dfffb4050 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:04.058893 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6ea7af-8f4b-4c26-ad3d-013e40b9623f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:04.077773 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:04.095440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eddb832-ccd0-4700-9b49-10e297e7529f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:04.095772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8952a1e-845e-4e03-b227-b5dd1a53abe9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:04.096369 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8952a1e-845e-4e03-b227-b5dd1a53abe9 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:04.096647 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8952a1e-845e-4e03-b227-b5dd1a53abe9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:04.096969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8952a1e-845e-4e03-b227-b5dd1a53abe9 None admin] PATH_INFO: `/v3/projects/26661cf5060b4b939d8ab02f9ea45daf` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:04.097254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eddb832-ccd0-4700-9b49-10e297e7529f None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:04.097731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eddb832-ccd0-4700-9b49-10e297e7529f None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:04.098187 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8952a1e-845e-4e03-b227-b5dd1a53abe9 None admin] RBAC: Authorizing `identity:get_project(project_id=26661cf5060b4b939d8ab02f9ea45daf)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:04.098367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eddb832-ccd0-4700-9b49-10e297e7529f None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:04.100118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eddb832-ccd0-4700-9b49-10e297e7529f None admin] RBAC: Adding query filter params (name=member) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:04.100222 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8952a1e-845e-4e03-b227-b5dd1a53abe9 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:04.100718 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eddb832-ccd0-4700-9b49-10e297e7529f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:04.102398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eddb832-ccd0-4700-9b49-10e297e7529f None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:04.108402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 214/431] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:13:03 2022] GET /identity/v3/projects/26661cf5060b4b939d8ab02f9ea45daf => generated 298 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:04.108402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8952a1e-845e-4e03-b227-b5dd1a53abe9 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:04.128015 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 218/432] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:13:03 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:04.129300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eddb832-ccd0-4700-9b49-10e297e7529f None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:04.158244 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:04.216009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8952a1e-845e-4e03-b227-b5dd1a53abe9 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:04.240390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eddb832-ccd0-4700-9b49-10e297e7529f None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:04.252865 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2598f69-662f-4c88-ab12-afaece5fc8b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:04.253446 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2598f69-662f-4c88-ab12-afaece5fc8b7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:04.253719 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2598f69-662f-4c88-ab12-afaece5fc8b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:04.253989 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2598f69-662f-4c88-ab12-afaece5fc8b7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:04.254715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2598f69-662f-4c88-ab12-afaece5fc8b7 None admin] RBAC: Adding query filter params (group.id=329d8769e54e48bc8acd9188320c782a, role.id=a9e862ac528245ca8363bccc5bf35287, scope.project.id=26661cf5060b4b939d8ab02f9ea45daf) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:04.255016 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2598f69-662f-4c88-ab12-afaece5fc8b7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:04.257005 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2598f69-662f-4c88-ab12-afaece5fc8b7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:04.263472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 215/433] 10.222.0.85 () {56 vars in 1376 bytes} [Tue Aug 30 04:13:04 2022] GET /identity/v3/role_assignments?group.id=329d8769e54e48bc8acd9188320c782a&scope.project.id=26661cf5060b4b939d8ab02f9ea45daf&role.id=a9e862ac528245ca8363bccc5bf35287 => generated 611 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:04.264288 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2598f69-662f-4c88-ab12-afaece5fc8b7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:04.280136 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2dcdc58-3533-486e-abbd-7267859a7a4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:04.281302 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dcdc58-3533-486e-abbd-7267859a7a4b None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:04.281836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dcdc58-3533-486e-abbd-7267859a7a4b None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:04.282306 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dcdc58-3533-486e-abbd-7267859a7a4b None admin] PATH_INFO: `/v3/groups/329d8769e54e48bc8acd9188320c782a` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:04.283778 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2dcdc58-3533-486e-abbd-7267859a7a4b None admin] RBAC: Authorizing `identity:get_group(group_id=329d8769e54e48bc8acd9188320c782a)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:04.285741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2dcdc58-3533-486e-abbd-7267859a7a4b None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:04.288526 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 219/434] 10.222.0.85 () {56 vars in 1180 bytes} [Tue Aug 30 04:13:04 2022] GET /identity/v3/groups/329d8769e54e48bc8acd9188320c782a => generated 226 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:04.298389 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2dcdc58-3533-486e-abbd-7267859a7a4b None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:04.327441 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:04.390058 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2598f69-662f-4c88-ab12-afaece5fc8b7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:04.430115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695fc185-c8f2-4e7a-88bb-95fd91e6732f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:04.430668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695fc185-c8f2-4e7a-88bb-95fd91e6732f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:04.431126 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695fc185-c8f2-4e7a-88bb-95fd91e6732f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:04.431417 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695fc185-c8f2-4e7a-88bb-95fd91e6732f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:04.497910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:04.510323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2dcdc58-3533-486e-abbd-7267859a7a4b None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:04.521425 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695fc185-c8f2-4e7a-88bb-95fd91e6732f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:04.523249 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695fc185-c8f2-4e7a-88bb-95fd91e6732f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:04.549777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e5459d-9474-4a99-b8f0-b815b156be36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:04.550382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e5459d-9474-4a99-b8f0-b815b156be36 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:04.550657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e5459d-9474-4a99-b8f0-b815b156be36 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:04.550925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e5459d-9474-4a99-b8f0-b815b156be36 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:04.551637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e5459d-9474-4a99-b8f0-b815b156be36 None admin] RBAC: Adding query filter params (group.id=329d8769e54e48bc8acd9188320c782a, role.id=ae0cef8120cf4dfc899c30803243a851, scope.project.id=26661cf5060b4b939d8ab02f9ea45daf) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:04.551938 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e5459d-9474-4a99-b8f0-b815b156be36 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:04.553438 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e5459d-9474-4a99-b8f0-b815b156be36 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:04.559119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 220/435] 10.222.0.85 () {56 vars in 1376 bytes} [Tue Aug 30 04:13:04 2022] GET /identity/v3/role_assignments?group.id=329d8769e54e48bc8acd9188320c782a&scope.project.id=26661cf5060b4b939d8ab02f9ea45daf&role.id=ae0cef8120cf4dfc899c30803243a851 => generated 611 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:04.560240 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234a08f1-5b09-4fd5-bae7-0990e8703420 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:04.560521 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234a08f1-5b09-4fd5-bae7-0990e8703420 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:04.560791 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234a08f1-5b09-4fd5-bae7-0990e8703420 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:04.562006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 221/436] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:04 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:04.563083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e566e0cb-cd42-4379-b3b6-dd83506713e5 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:04.563362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e566e0cb-cd42-4379-b3b6-dd83506713e5 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:04.563630 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e566e0cb-cd42-4379-b3b6-dd83506713e5 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:04.565059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 222/437] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:04 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:04.570456 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c19e613-fa70-4629-bcad-03418fa3ab0f None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:04.570738 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c19e613-fa70-4629-bcad-03418fa3ab0f None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:04.571014 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c19e613-fa70-4629-bcad-03418fa3ab0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:04.591865 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:04.592065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-6c19e613-fa70-4629-bcad-03418fa3ab0f None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:04.593279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-6c19e613-fa70-4629-bcad-03418fa3ab0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:04.658832 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 216/438] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:13:04.658832 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b32e7fe-0af6-404e-9858-29e0fb796dfe None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:04.658832 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b32e7fe-0af6-404e-9858-29e0fb796dfe None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:04.658832 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b32e7fe-0af6-404e-9858-29e0fb796dfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:04.681191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 223/439] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:04.681191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c19e613-fa70-4629-bcad-03418fa3ab0f None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:04.681789 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4b32e7fe-0af6-404e-9858-29e0fb796dfe None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:04.681789 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4b32e7fe-0af6-404e-9858-29e0fb796dfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:04.763274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 217/440] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:04.785934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:04.804558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c19e613-fa70-4629-bcad-03418fa3ab0f None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:04.839341 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4391d24-6e51-4ff7-9048-d382bc5a83af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:04.839972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4391d24-6e51-4ff7-9048-d382bc5a83af None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:04.840246 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4391d24-6e51-4ff7-9048-d382bc5a83af None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:04.840512 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4391d24-6e51-4ff7-9048-d382bc5a83af None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:04.841742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4391d24-6e51-4ff7-9048-d382bc5a83af None admin] RBAC: Authorizing `identity:get_project(project_id=008c6a0a4a114009bc7c692aabd0b0c8)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:04.843214 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4391d24-6e51-4ff7-9048-d382bc5a83af None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:04.845168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 224/441] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:13:04 2022] GET /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8 => generated 302 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:04.851004 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b32e7fe-0af6-404e-9858-29e0fb796dfe None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:04.978618 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:05.011040 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b32e7fe-0af6-404e-9858-29e0fb796dfe None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:05.033800 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4391d24-6e51-4ff7-9048-d382bc5a83af None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:05.059693 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71e2b0e-100c-48b1-8f90-9677790fc7fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:05.060332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71e2b0e-100c-48b1-8f90-9677790fc7fb None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:05.061196 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71e2b0e-100c-48b1-8f90-9677790fc7fb None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:05.061196 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71e2b0e-100c-48b1-8f90-9677790fc7fb None admin] PATH_INFO: `/v3/groups/329d8769e54e48bc8acd9188320c782a` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:05.062508 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71e2b0e-100c-48b1-8f90-9677790fc7fb None admin] RBAC: Authorizing `identity:get_group(group_id=329d8769e54e48bc8acd9188320c782a)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:05.063968 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71e2b0e-100c-48b1-8f90-9677790fc7fb None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:05.065640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 218/442] 10.222.0.85 () {56 vars in 1180 bytes} [Tue Aug 30 04:13:04 2022] GET /identity/v3/groups/329d8769e54e48bc8acd9188320c782a => generated 226 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:05.069796 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71e2b0e-100c-48b1-8f90-9677790fc7fb None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:05.173314 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4391d24-6e51-4ff7-9048-d382bc5a83af None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:05.187335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71e2b0e-100c-48b1-8f90-9677790fc7fb None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:05.207383 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0348b3db-ec0f-4ab8-9ffd-1ae9f0eaa5ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:05.208016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0348b3db-ec0f-4ab8-9ffd-1ae9f0eaa5ef None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:05.208314 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0348b3db-ec0f-4ab8-9ffd-1ae9f0eaa5ef None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:05.209110 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0348b3db-ec0f-4ab8-9ffd-1ae9f0eaa5ef None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:05.218962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0348b3db-ec0f-4ab8-9ffd-1ae9f0eaa5ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:05.220655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751edc43-c171-434b-8b73-140efb48d48f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:05.221312 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751edc43-c171-434b-8b73-140efb48d48f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:05.221591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751edc43-c171-434b-8b73-140efb48d48f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:05.221863 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751edc43-c171-434b-8b73-140efb48d48f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:05.223876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-0348b3db-ec0f-4ab8-9ffd-1ae9f0eaa5ef None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:13:05.223876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:05.223876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:05.223876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:05.223876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:05.223876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:05.223876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:05.223876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:05.223876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:05.223876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:05.223876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:05.223876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:05.223876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:05.223876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:05.223876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:05.223876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:05.225409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:05.225409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:05.225409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:05.225409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:05.225409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:05.225409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:05.225409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:05.225409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:05.225409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:05.225409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:05.225409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:05.225409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:05.225409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:05.225409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:05.225409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:05.225409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:05.226200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:05.226200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:05.226200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:05.226200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:05.226200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:05.226200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:05.226200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:05.226200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:05.226200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:05.226200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:05.226200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:13:05.226200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:05.226200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 225/443] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:13:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:05.230125 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0348b3db-ec0f-4ab8-9ffd-1ae9f0eaa5ef None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:05.233129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751edc43-c171-434b-8b73-140efb48d48f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:05.238010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-751edc43-c171-434b-8b73-140efb48d48f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:13:05.238010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:05.238010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:05.238010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:05.238010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:05.238010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:05.238010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:05.238010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:05.238010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:05.238010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:05.238010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:05.238010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:05.238010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:05.238010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:05.238010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:05.238010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:05.238998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:05.238998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:05.238998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:05.238998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:05.238998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:05.238998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:05.238998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:05.238998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:05.238998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:05.238998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:05.238998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:05.238998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:05.238998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:05.238998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:05.238998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:05.238998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:05.239866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:05.239866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:05.239866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:05.239866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:05.239866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:05.239866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:05.239866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:05.239866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:05.239866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:05.239866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:05.239866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:13:05.239866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:05.239866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 219/444] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:13:05 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:05.240928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751edc43-c171-434b-8b73-140efb48d48f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:05.250879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:05.252042 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:05.337883 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0348b3db-ec0f-4ab8-9ffd-1ae9f0eaa5ef None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:05.341097 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751edc43-c171-434b-8b73-140efb48d48f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:05.372428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eada5096-dd41-4517-b78a-051d6e86f5dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:05.373070 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eada5096-dd41-4517-b78a-051d6e86f5dc None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:05.373340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eada5096-dd41-4517-b78a-051d6e86f5dc None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:05.373610 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eada5096-dd41-4517-b78a-051d6e86f5dc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:05.374330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eada5096-dd41-4517-b78a-051d6e86f5dc None admin] RBAC: Adding query filter params (group.id=329d8769e54e48bc8acd9188320c782a, role.id=ae0cef8120cf4dfc899c30803243a851, scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:05.374636 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eada5096-dd41-4517-b78a-051d6e86f5dc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:05.375119 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bceb99d-e45a-4596-b059-1829584ea8cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:05.375724 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bceb99d-e45a-4596-b059-1829584ea8cd None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:05.376006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bceb99d-e45a-4596-b059-1829584ea8cd None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:05.376092 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eada5096-dd41-4517-b78a-051d6e86f5dc None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:05.376279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bceb99d-e45a-4596-b059-1829584ea8cd None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:05.376940 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bceb99d-e45a-4596-b059-1829584ea8cd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:05.377275 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bceb99d-e45a-4596-b059-1829584ea8cd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:05.378961 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bceb99d-e45a-4596-b059-1829584ea8cd None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:05.382538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 226/445] 10.222.0.85 () {56 vars in 1376 bytes} [Tue Aug 30 04:13:05 2022] GET /identity/v3/role_assignments?group.id=329d8769e54e48bc8acd9188320c782a&scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8&role.id=ae0cef8120cf4dfc899c30803243a851 => generated 611 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:05.383319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eada5096-dd41-4517-b78a-051d6e86f5dc None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:05.392458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 220/446] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:13:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:05.393683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bceb99d-e45a-4596-b059-1829584ea8cd None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:05.476213 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:05.479034 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:05.505965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eada5096-dd41-4517-b78a-051d6e86f5dc None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:05.512017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bceb99d-e45a-4596-b059-1829584ea8cd None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:05.548664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04043d8-6969-4b9f-bd89-dcca99b55daa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:05.548887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628728b1-c1e4-4e2d-b5e5-0363b2b5f43c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:05.549316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04043d8-6969-4b9f-bd89-dcca99b55daa None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:05.549497 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628728b1-c1e4-4e2d-b5e5-0363b2b5f43c None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:05.549590 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04043d8-6969-4b9f-bd89-dcca99b55daa None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:05.549740 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628728b1-c1e4-4e2d-b5e5-0363b2b5f43c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:05.549900 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04043d8-6969-4b9f-bd89-dcca99b55daa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:05.550043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628728b1-c1e4-4e2d-b5e5-0363b2b5f43c None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:05.550932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628728b1-c1e4-4e2d-b5e5-0363b2b5f43c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:05.551245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628728b1-c1e4-4e2d-b5e5-0363b2b5f43c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:05.554213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628728b1-c1e4-4e2d-b5e5-0363b2b5f43c None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:05.563158 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 227/447] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:05 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:05.568954 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628728b1-c1e4-4e2d-b5e5-0363b2b5f43c None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:05.633170 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04043d8-6969-4b9f-bd89-dcca99b55daa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:05.634811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04043d8-6969-4b9f-bd89-dcca99b55daa None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:05.687492 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628728b1-c1e4-4e2d-b5e5-0363b2b5f43c None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:05.715540 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:05.730313 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a480fb-a6ab-4410-acce-7ab6b0ba1804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:05.730938 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a480fb-a6ab-4410-acce-7ab6b0ba1804 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:05.731219 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a480fb-a6ab-4410-acce-7ab6b0ba1804 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:05.731492 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a480fb-a6ab-4410-acce-7ab6b0ba1804 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:05.743847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:05.752681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 221/448] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:13:05.758710 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04043d8-6969-4b9f-bd89-dcca99b55daa None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:05.807830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a480fb-a6ab-4410-acce-7ab6b0ba1804 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:05.809306 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a480fb-a6ab-4410-acce-7ab6b0ba1804 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:05.855479 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04043d8-6969-4b9f-bd89-dcca99b55daa None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:05.889586 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0dc5c3-523b-4d82-a51f-9fc3be31b100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:05.890203 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0dc5c3-523b-4d82-a51f-9fc3be31b100 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:05.890476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0dc5c3-523b-4d82-a51f-9fc3be31b100 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:05.890743 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0dc5c3-523b-4d82-a51f-9fc3be31b100 None admin] PATH_INFO: `/v3/projects/be2e356bbe7d4680bd52a62e89c77545` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:05.891839 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0dc5c3-523b-4d82-a51f-9fc3be31b100 None admin] RBAC: Authorizing `identity:get_project(project_id=be2e356bbe7d4680bd52a62e89c77545)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:05.893290 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0dc5c3-523b-4d82-a51f-9fc3be31b100 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:05.895045 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 222/449] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:13:05 2022] GET /identity/v3/projects/be2e356bbe7d4680bd52a62e89c77545 => generated 344 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:05.900480 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0dc5c3-523b-4d82-a51f-9fc3be31b100 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:05.911467 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 228/450] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:13:05.912796 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a480fb-a6ab-4410-acce-7ab6b0ba1804 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:05.939814 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:05.953738 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:06.000675 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0dc5c3-523b-4d82-a51f-9fc3be31b100 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:06.012275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a480fb-a6ab-4410-acce-7ab6b0ba1804 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:06.034420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bc4b3a-aa6f-4575-a179-865f77116fce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:06.035059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bc4b3a-aa6f-4575-a179-865f77116fce None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:06.035336 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bc4b3a-aa6f-4575-a179-865f77116fce None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:06.035614 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bc4b3a-aa6f-4575-a179-865f77116fce None admin] PATH_INFO: `/v3/groups/7f39b97c2dba444da6af6f3b248d96ad` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:06.037353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bc4b3a-aa6f-4575-a179-865f77116fce None admin] RBAC: Authorizing `identity:get_group(group_id=7f39b97c2dba444da6af6f3b248d96ad)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:06.038803 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bc4b3a-aa6f-4575-a179-865f77116fce None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:06.040379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 223/451] 10.222.0.85 () {56 vars in 1180 bytes} [Tue Aug 30 04:13:05 2022] GET /identity/v3/groups/7f39b97c2dba444da6af6f3b248d96ad => generated 229 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:06.046184 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc108aec-762c-498e-b111-8a554fa4fb02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:06.046574 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bc4b3a-aa6f-4575-a179-865f77116fce None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:06.046807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc108aec-762c-498e-b111-8a554fa4fb02 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:06.047084 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc108aec-762c-498e-b111-8a554fa4fb02 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:06.047351 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc108aec-762c-498e-b111-8a554fa4fb02 None admin] PATH_INFO: `/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:06.048445 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc108aec-762c-498e-b111-8a554fa4fb02 None admin] RBAC: Authorizing `identity:get_project(project_id=008c6a0a4a114009bc7c692aabd0b0c8)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:06.049906 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc108aec-762c-498e-b111-8a554fa4fb02 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:06.051693 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 229/452] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:13:05 2022] GET /identity/v3/projects/008c6a0a4a114009bc7c692aabd0b0c8 => generated 302 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:06.057080 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc108aec-762c-498e-b111-8a554fa4fb02 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:06.150116 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bc4b3a-aa6f-4575-a179-865f77116fce None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:06.157302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:06.161375 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc108aec-762c-498e-b111-8a554fa4fb02 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:06.172363 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:06.185365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5aa98f-df1e-4189-9703-5811456d1995 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:06.185974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5aa98f-df1e-4189-9703-5811456d1995 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:06.186257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5aa98f-df1e-4189-9703-5811456d1995 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:06.186531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5aa98f-df1e-4189-9703-5811456d1995 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:06.187257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5aa98f-df1e-4189-9703-5811456d1995 None admin] RBAC: Adding query filter params (group.id=7f39b97c2dba444da6af6f3b248d96ad, role.id=d17954e7ce544f0d90a512340232c60e, scope.project.id=be2e356bbe7d4680bd52a62e89c77545) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:06.187565 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5aa98f-df1e-4189-9703-5811456d1995 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:06.188999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5aa98f-df1e-4189-9703-5811456d1995 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:06.194957 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 224/453] 10.222.0.85 () {56 vars in 1376 bytes} [Tue Aug 30 04:13:06 2022] GET /identity/v3/role_assignments?group.id=7f39b97c2dba444da6af6f3b248d96ad&scope.project.id=be2e356bbe7d4680bd52a62e89c77545&role.id=d17954e7ce544f0d90a512340232c60e => generated 611 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:06.203868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9207b0-528e-4b7b-b6ff-1f832038c73f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:06.204526 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9207b0-528e-4b7b-b6ff-1f832038c73f None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:06.204802 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9207b0-528e-4b7b-b6ff-1f832038c73f None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:06.205094 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9207b0-528e-4b7b-b6ff-1f832038c73f None admin] PATH_INFO: `/v3/groups/329d8769e54e48bc8acd9188320c782a` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:06.206268 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9207b0-528e-4b7b-b6ff-1f832038c73f None admin] RBAC: Authorizing `identity:get_group(group_id=329d8769e54e48bc8acd9188320c782a)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:06.207749 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9207b0-528e-4b7b-b6ff-1f832038c73f None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:06.209992 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 230/454] 10.222.0.85 () {56 vars in 1180 bytes} [Tue Aug 30 04:13:06 2022] GET /identity/v3/groups/329d8769e54e48bc8acd9188320c782a => generated 226 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:06.216017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5aa98f-df1e-4189-9703-5811456d1995 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:06.321314 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5aa98f-df1e-4189-9703-5811456d1995 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:06.358201 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c9e0a5-b50f-43e2-8bd5-9de9e0918144 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:06.358816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c9e0a5-b50f-43e2-8bd5-9de9e0918144 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:06.359106 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c9e0a5-b50f-43e2-8bd5-9de9e0918144 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:06.359372 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c9e0a5-b50f-43e2-8bd5-9de9e0918144 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:06.360104 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c9e0a5-b50f-43e2-8bd5-9de9e0918144 None admin] RBAC: Adding query filter params (group.id=329d8769e54e48bc8acd9188320c782a, role.id=a9e862ac528245ca8363bccc5bf35287, scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:06.360407 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c9e0a5-b50f-43e2-8bd5-9de9e0918144 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:06.361863 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c9e0a5-b50f-43e2-8bd5-9de9e0918144 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:06.368363 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 225/455] 10.222.0.85 () {56 vars in 1376 bytes} [Tue Aug 30 04:13:06 2022] GET /identity/v3/role_assignments?group.id=329d8769e54e48bc8acd9188320c782a&scope.project.id=008c6a0a4a114009bc7c692aabd0b0c8&role.id=a9e862ac528245ca8363bccc5bf35287 => generated 611 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:08.951197 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2ed303-3d76-4ae5-a665-e6838aae3317 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:08.951774 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2ed303-3d76-4ae5-a665-e6838aae3317 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:08.951774 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2ed303-3d76-4ae5-a665-e6838aae3317 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:08.953361 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 231/456] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:08.958907 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3713ddc4-3ca3-4209-b36a-609afc4d402c None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:08.959359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3713ddc4-3ca3-4209-b36a-609afc4d402c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:08.959451 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3713ddc4-3ca3-4209-b36a-609afc4d402c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:08.981639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3713ddc4-3ca3-4209-b36a-609afc4d402c None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:08.982839 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3713ddc4-3ca3-4209-b36a-609afc4d402c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:09.024221 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:09.047393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fec1bb1-2240-49f3-bb77-6bfe24eed5c6 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:09.047720 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fec1bb1-2240-49f3-bb77-6bfe24eed5c6 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:09.047949 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fec1bb1-2240-49f3-bb77-6bfe24eed5c6 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:09.049557 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 232/457] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:09.055143 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96ec73c-2678-4a70-bafc-06060adac0db None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:09.055424 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96ec73c-2678-4a70-bafc-06060adac0db None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:09.055699 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96ec73c-2678-4a70-bafc-06060adac0db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:09.075493 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-d96ec73c-2678-4a70-bafc-06060adac0db None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:09.077023 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d96ec73c-2678-4a70-bafc-06060adac0db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:09.086792 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 226/458] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:09.159997 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 233/459] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:13:09.199272 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89013232-bf50-4ada-b071-6cdc1b654616 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:09.199785 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89013232-bf50-4ada-b071-6cdc1b654616 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:09.200195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89013232-bf50-4ada-b071-6cdc1b654616 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:09.201786 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 227/460] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:09.207016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12235b3e-c244-4dc8-8723-b87648bb1d0b None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:09.207483 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12235b3e-c244-4dc8-8723-b87648bb1d0b None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:09.207896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12235b3e-c244-4dc8-8723-b87648bb1d0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:09.238010 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-12235b3e-c244-4dc8-8723-b87648bb1d0b None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:09.239417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-12235b3e-c244-4dc8-8723-b87648bb1d0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:09.323250 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 234/461] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:09.480376 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89013232-bf50-4ada-b071-6cdc1b654616 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:09.525418 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12235b3e-c244-4dc8-8723-b87648bb1d0b None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:09.583744 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:09.598317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89013232-bf50-4ada-b071-6cdc1b654616 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:09.642842 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ce126c-ee7b-41d2-b9a9-b3f8108bf616 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:09.643964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ce126c-ee7b-41d2-b9a9-b3f8108bf616 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:09.644486 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ce126c-ee7b-41d2-b9a9-b3f8108bf616 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:09.644949 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ce126c-ee7b-41d2-b9a9-b3f8108bf616 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:09.657647 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ce126c-ee7b-41d2-b9a9-b3f8108bf616 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:09.659842 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ce126c-ee7b-41d2-b9a9-b3f8108bf616 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:09.672273 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12235b3e-c244-4dc8-8723-b87648bb1d0b None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:09.676010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-91ce126c-ee7b-41d2-b9a9-b3f8108bf616 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:09.676010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:09.676010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:09.676010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:09.676010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:09.676010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:09.676010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:09.676010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:09.676010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:09.676010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:09.676010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:09.676010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:09.676010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:09.676010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:09.676010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:09.676010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:09.676871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:09.676871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:09.676871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:09.676871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:09.676871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:09.676871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:09.676871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:09.676871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:09.676871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:09.676871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:09.676871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:09.676871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:09.676871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:09.676871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:09.676871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:09.676871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:09.676871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:09.677889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:09.677889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:09.677889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:09.683257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 228/462] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:09.683257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbe41df-d5bc-4068-83d6-7939497c5af9 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:09.683257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbe41df-d5bc-4068-83d6-7939497c5af9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:09.683257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbe41df-d5bc-4068-83d6-7939497c5af9 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:09.683257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 229/463] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:09.684227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbe41df-d5bc-4068-83d6-7939497c5af9 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:09.702390 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:09.715273 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485a4afc-7dc3-417b-8398-e3f062d051cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:09.715981 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485a4afc-7dc3-417b-8398-e3f062d051cc None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:09.716324 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485a4afc-7dc3-417b-8398-e3f062d051cc None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:09.716660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485a4afc-7dc3-417b-8398-e3f062d051cc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:09.727777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485a4afc-7dc3-417b-8398-e3f062d051cc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:09.729258 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485a4afc-7dc3-417b-8398-e3f062d051cc None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:09.745394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-485a4afc-7dc3-417b-8398-e3f062d051cc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:09.745394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:09.745394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:09.745394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:09.745394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:09.745394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:09.745394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:09.745394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:09.745394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:09.745394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:09.745394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:09.745394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:09.745394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:09.745394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:09.745394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:09.745394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:09.746439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:09.746439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:09.746439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:09.746439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:09.746439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:09.746439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:09.746439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:09.746439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:09.746439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:09.746439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:09.746439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:09.746439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:09.746439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:09.746439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:09.746439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:09.746439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:09.746439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:09.747569 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:09.747569 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:09.747569 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:09.748917 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 235/464] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:09.749859 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485a4afc-7dc3-417b-8398-e3f062d051cc None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:09.799064 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbe41df-d5bc-4068-83d6-7939497c5af9 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:09.834120 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa6a866-9f91-46cc-8e03-e4445053fd2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:09.835005 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa6a866-9f91-46cc-8e03-e4445053fd2a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:09.835441 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa6a866-9f91-46cc-8e03-e4445053fd2a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:09.835869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa6a866-9f91-46cc-8e03-e4445053fd2a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:09.842920 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:09.847193 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa6a866-9f91-46cc-8e03-e4445053fd2a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:09.848924 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa6a866-9f91-46cc-8e03-e4445053fd2a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:09.857503 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485a4afc-7dc3-417b-8398-e3f062d051cc None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:09.866676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-aaa6a866-9f91-46cc-8e03-e4445053fd2a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:09.866676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:09.866676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:09.866676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:09.866676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:09.866676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:09.866676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:09.866676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:09.866676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:09.866676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:09.866676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:09.866676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:09.866676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:09.866676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:09.866676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:09.866676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:09.867741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:09.867741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:09.867741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:09.867741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:09.867741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:09.867741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:09.867741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:09.867741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:09.867741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:09.867741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:09.867741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:09.867741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:09.867741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:09.867741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:09.867741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:09.867741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:09.867741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:09.868619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:09.868619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:09.868619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:09.869356 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 230/465] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:09.870383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16bd8f9-5276-4a9e-81fc-d3bc32c64f07 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:09.870670 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16bd8f9-5276-4a9e-81fc-d3bc32c64f07 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:09.870945 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16bd8f9-5276-4a9e-81fc-d3bc32c64f07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:09.889569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-e16bd8f9-5276-4a9e-81fc-d3bc32c64f07 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:09.890801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e16bd8f9-5276-4a9e-81fc-d3bc32c64f07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:09.898849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c78068d-3a50-40a0-9207-a31ccb286688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:09.899444 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c78068d-3a50-40a0-9207-a31ccb286688 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:09.899722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c78068d-3a50-40a0-9207-a31ccb286688 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:09.899988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c78068d-3a50-40a0-9207-a31ccb286688 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:09.900677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c78068d-3a50-40a0-9207-a31ccb286688 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:09.900916 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c78068d-3a50-40a0-9207-a31ccb286688 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:09.902389 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c78068d-3a50-40a0-9207-a31ccb286688 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:09.919669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 236/466] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:09.920570 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c78068d-3a50-40a0-9207-a31ccb286688 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:09.956893 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:09.964081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 231/467] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:09.966284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16bd8f9-5276-4a9e-81fc-d3bc32c64f07 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:10.025349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c78068d-3a50-40a0-9207-a31ccb286688 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:10.062216 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253d1a9b-2cab-4cbf-a919-4904f5d93d05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:10.063146 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253d1a9b-2cab-4cbf-a919-4904f5d93d05 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:10.063575 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253d1a9b-2cab-4cbf-a919-4904f5d93d05 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:10.064035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253d1a9b-2cab-4cbf-a919-4904f5d93d05 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:10.064825 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253d1a9b-2cab-4cbf-a919-4904f5d93d05 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:10.065300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253d1a9b-2cab-4cbf-a919-4904f5d93d05 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:10.066834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253d1a9b-2cab-4cbf-a919-4904f5d93d05 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:10.079144 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 237/468] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:10.080273 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253d1a9b-2cab-4cbf-a919-4904f5d93d05 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:10.097406 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:10.098259 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16bd8f9-5276-4a9e-81fc-d3bc32c64f07 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:10.144482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc079222-b3f3-47cd-a1a9-db28220d1fe1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:10.145277 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc079222-b3f3-47cd-a1a9-db28220d1fe1 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:10.145666 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc079222-b3f3-47cd-a1a9-db28220d1fe1 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:10.146047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc079222-b3f3-47cd-a1a9-db28220d1fe1 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:10.147195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc079222-b3f3-47cd-a1a9-db28220d1fe1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:10.147578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc079222-b3f3-47cd-a1a9-db28220d1fe1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:10.149324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc079222-b3f3-47cd-a1a9-db28220d1fe1 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:10.173520 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 232/469] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:10.174890 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc079222-b3f3-47cd-a1a9-db28220d1fe1 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:10.200319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253d1a9b-2cab-4cbf-a919-4904f5d93d05 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:10.236789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43005f0-0450-4ac9-8cd8-b8c5fe671d37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:10.237701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43005f0-0450-4ac9-8cd8-b8c5fe671d37 None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:10.238170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43005f0-0450-4ac9-8cd8-b8c5fe671d37 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:10.238640 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43005f0-0450-4ac9-8cd8-b8c5fe671d37 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:10.239600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43005f0-0450-4ac9-8cd8-b8c5fe671d37 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:10.241823 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43005f0-0450-4ac9-8cd8-b8c5fe671d37 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:10.251918 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:10.267242 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 238/470] 10.222.0.85 () {60 vars in 1132 bytes} [Tue Aug 30 04:13:10 2022] POST /identity/v3/users => generated 248 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:13:10.268304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43005f0-0450-4ac9-8cd8-b8c5fe671d37 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:10.303584 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc079222-b3f3-47cd-a1a9-db28220d1fe1 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:10.344997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba46674-5081-447b-aba6-8231692f39af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:10.346092 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba46674-5081-447b-aba6-8231692f39af None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:10.346556 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba46674-5081-447b-aba6-8231692f39af None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:10.347011 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba46674-5081-447b-aba6-8231692f39af None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:10.347879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba46674-5081-447b-aba6-8231692f39af None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:10.350299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba46674-5081-447b-aba6-8231692f39af None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:10.370811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:10.376824 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 233/471] 10.222.0.85 () {60 vars in 1132 bytes} [Tue Aug 30 04:13:10 2022] POST /identity/v3/users => generated 246 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:13:10.377807 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba46674-5081-447b-aba6-8231692f39af None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:10.425282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43005f0-0450-4ac9-8cd8-b8c5fe671d37 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:10.470742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3072064b-b546-450b-b719-fdcf74ad10bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:10.471357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3072064b-b546-450b-b719-fdcf74ad10bd None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:10.471642 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3072064b-b546-450b-b719-fdcf74ad10bd None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:10.471941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3072064b-b546-450b-b719-fdcf74ad10bd None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:10.472920 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3072064b-b546-450b-b719-fdcf74ad10bd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:10.474315 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3072064b-b546-450b-b719-fdcf74ad10bd None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:10.501770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 239/472] 10.222.0.85 () {60 vars in 1132 bytes} [Tue Aug 30 04:13:10 2022] POST /identity/v3/users => generated 248 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:13:10.503366 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a14ed26-1117-405b-be17-2cf8f66b782f None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:10.503653 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a14ed26-1117-405b-be17-2cf8f66b782f None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:10.503924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a14ed26-1117-405b-be17-2cf8f66b782f None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:10.509875 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 240/473] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:10.512178 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9261fd03-1a5e-456f-b98e-8bff4322a231 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:10.512719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9261fd03-1a5e-456f-b98e-8bff4322a231 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:10.513171 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9261fd03-1a5e-456f-b98e-8bff4322a231 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:10.534518 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba46674-5081-447b-aba6-8231692f39af None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:10.544753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-9261fd03-1a5e-456f-b98e-8bff4322a231 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:10.547211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-9261fd03-1a5e-456f-b98e-8bff4322a231 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:10.556634 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:10.592251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78adb2b2-ac20-4358-8979-fc7477e05e2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:10.593317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78adb2b2-ac20-4358-8979-fc7477e05e2e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:10.593602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78adb2b2-ac20-4358-8979-fc7477e05e2e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:10.593876 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78adb2b2-ac20-4358-8979-fc7477e05e2e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:10.608653 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78adb2b2-ac20-4358-8979-fc7477e05e2e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:10.610557 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78adb2b2-ac20-4358-8979-fc7477e05e2e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:10.631538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-78adb2b2-ac20-4358-8979-fc7477e05e2e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:10.631538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:10.631538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:10.631538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:10.631538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:10.631538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:10.631538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:10.631538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:10.631538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:10.631538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:10.631538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:10.631538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:10.631538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:10.631538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:10.631538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:10.631538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:10.632507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:10.632507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:10.632507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:10.632507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:10.632507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:10.632507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:10.632507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:10.632507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:10.632507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:10.632507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:10.632507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:10.632507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:10.632507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:10.632507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:10.632507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:10.632507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:10.632507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:10.634063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:10.634063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:10.634063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:10.634063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 234/474] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:10.639114 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78adb2b2-ac20-4358-8979-fc7477e05e2e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:10.645858 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 241/475] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:10.746528 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78adb2b2-ac20-4358-8979-fc7477e05e2e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:10.782587 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ed6636-33f6-494a-aa7e-d14c8d42e17d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:10.783202 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ed6636-33f6-494a-aa7e-d14c8d42e17d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:10.783503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ed6636-33f6-494a-aa7e-d14c8d42e17d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:10.783754 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ed6636-33f6-494a-aa7e-d14c8d42e17d None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:10.784403 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ed6636-33f6-494a-aa7e-d14c8d42e17d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:10.784699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ed6636-33f6-494a-aa7e-d14c8d42e17d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:10.786108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ed6636-33f6-494a-aa7e-d14c8d42e17d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:10.799862 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 235/476] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:10.805839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9261fd03-1a5e-456f-b98e-8bff4322a231 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:10.859337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:10.914097 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9261fd03-1a5e-456f-b98e-8bff4322a231 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:10.934069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ed6636-33f6-494a-aa7e-d14c8d42e17d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:10.956603 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d89559-9ffa-43f7-b3ea-a4b5fac4858c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:10.957272 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d89559-9ffa-43f7-b3ea-a4b5fac4858c None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:10.957550 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d89559-9ffa-43f7-b3ea-a4b5fac4858c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:10.957821 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d89559-9ffa-43f7-b3ea-a4b5fac4858c None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:10.959055 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d89559-9ffa-43f7-b3ea-a4b5fac4858c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:10.960561 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d89559-9ffa-43f7-b3ea-a4b5fac4858c None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:10.968386 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:10.988258 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 242/477] 10.222.0.85 () {60 vars in 1132 bytes} [Tue Aug 30 04:13:10 2022] POST /identity/v3/users => generated 249 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:13:11.085149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ed6636-33f6-494a-aa7e-d14c8d42e17d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:11.136323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db25627-c83a-4563-8549-dfd01c554e64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:11.136954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db25627-c83a-4563-8549-dfd01c554e64 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:11.137320 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db25627-c83a-4563-8549-dfd01c554e64 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:11.137593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db25627-c83a-4563-8549-dfd01c554e64 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:11.138274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db25627-c83a-4563-8549-dfd01c554e64 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:11.139670 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db25627-c83a-4563-8549-dfd01c554e64 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:11.150216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.sql.core [None req-4db25627-c83a-4563-8549-dfd01c554e64 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 30 04:13:11.150216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 30 04:13:11.150216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [parameters: {'id': '2ae295d050c14194b8b66cf592907895', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 30 04:13:11.150216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81485) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 30 04:13:11.167744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-4db25627-c83a-4563-8549-dfd01c554e64 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 30 04:13:11.167744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:11.167744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 30 04:13:11.167744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 30 04:13:11.167744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 30 04:13:11.167744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 30 04:13:11.167744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 30 04:13:11.167744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result = self._query(query) Aug 30 04:13:11.167744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 30 04:13:11.167744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application conn.query(q) Aug 30 04:13:11.167744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 30 04:13:11.167744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 30 04:13:11.167744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 30 04:13:11.167744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result.read() Aug 30 04:13:11.167744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 30 04:13:11.168926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 30 04:13:11.168926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 30 04:13:11.168926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 30 04:13:11.168926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 30 04:13:11.168926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 30 04:13:11.168926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 30 04:13:11.168926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 30 04:13:11.168926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 30 04:13:11.168926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:11.168926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 30 04:13:11.168926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:11.168926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:11.168926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 30 04:13:11.168926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 30 04:13:11.168926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Aug 30 04:13:11.168926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return ref.to_dict() Aug 30 04:13:11.168926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 30 04:13:11.168926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application next(self.gen) Aug 30 04:13:11.169866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 30 04:13:11.169866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application yield resource Aug 30 04:13:11.169866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 30 04:13:11.169866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application next(self.gen) Aug 30 04:13:11.169866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 30 04:13:11.169866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.session.rollback() Aug 30 04:13:11.169866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 30 04:13:11.169866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.force_reraise() Aug 30 04:13:11.169866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 30 04:13:11.169866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise self.value Aug 30 04:13:11.169866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 30 04:13:11.169866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 30 04:13:11.169866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 30 04:13:11.169866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application session.commit() Aug 30 04:13:11.169866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 30 04:13:11.169866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 30 04:13:11.169866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 30 04:13:11.170736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._prepare_impl() Aug 30 04:13:11.170736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 30 04:13:11.170736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.session.flush() Aug 30 04:13:11.170736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 30 04:13:11.170736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._flush(objects) Aug 30 04:13:11.170736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 30 04:13:11.170736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 30 04:13:11.170736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 30 04:13:11.170736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application compat.raise_( Aug 30 04:13:11.170736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 30 04:13:11.170736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception Aug 30 04:13:11.170736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 30 04:13:11.170736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application flush_context.execute() Aug 30 04:13:11.170736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 30 04:13:11.170736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rec.execute(self) Aug 30 04:13:11.170736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 30 04:13:11.170736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 30 04:13:11.171596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 30 04:13:11.171596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 30 04:13:11.171596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 30 04:13:11.171596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 30 04:13:11.171596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 30 04:13:11.171596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 30 04:13:11.171596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 30 04:13:11.171596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 30 04:13:11.171596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 30 04:13:11.171596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 30 04:13:11.171596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 30 04:13:11.171596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 30 04:13:11.171596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 30 04:13:11.171596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 30 04:13:11.171596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 30 04:13:11.171596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception Aug 30 04:13:11.172420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 30 04:13:11.172420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 30 04:13:11.172420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 30 04:13:11.172420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 30 04:13:11.172420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 30 04:13:11.172420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result = self._query(query) Aug 30 04:13:11.172420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 30 04:13:11.172420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application conn.query(q) Aug 30 04:13:11.172420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 30 04:13:11.172420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 30 04:13:11.172420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 30 04:13:11.172420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result.read() Aug 30 04:13:11.172420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 30 04:13:11.172420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 30 04:13:11.172420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 30 04:13:11.172420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 30 04:13:11.173236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 30 04:13:11.173236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 30 04:13:11.173236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 30 04:13:11.173236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 30 04:13:11.173236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 30 04:13:11.173236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 30 04:13:11.173236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [parameters: {'id': '2ae295d050c14194b8b66cf592907895', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 30 04:13:11.173236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 30 04:13:11.173236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:11.173236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:13:11.173236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:11.173236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:11.173236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:11.173236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:11.173236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:11.173236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:11.174047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:11.174047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:11.174047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:11.174047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:11.174047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:11.174047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:11.174047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Aug 30 04:13:11.174047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Aug 30 04:13:11.174047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:11.174047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:11.174047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Aug 30 04:13:11.174047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Aug 30 04:13:11.174047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 30 04:13:11.174047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 30 04:13:11.174047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 30 04:13:11.174047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:11.174898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 236/478] 10.222.0.85 () {60 vars in 1131 bytes} [Tue Aug 30 04:13:10 2022] POST /identity/v3/roles => generated 144 bytes in 236 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:11.178926 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d89559-9ffa-43f7-b3ea-a4b5fac4858c None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:11.295756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d89559-9ffa-43f7-b3ea-a4b5fac4858c None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:11.305061 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:11.337245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70203dbb-d87a-4b10-b2b9-83dd395e1ecd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:11.337924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70203dbb-d87a-4b10-b2b9-83dd395e1ecd None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:11.338205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70203dbb-d87a-4b10-b2b9-83dd395e1ecd None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:11.338528 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70203dbb-d87a-4b10-b2b9-83dd395e1ecd None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:11.351522 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70203dbb-d87a-4b10-b2b9-83dd395e1ecd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:11.356775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-70203dbb-d87a-4b10-b2b9-83dd395e1ecd None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:13:11.356775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:11.356775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:11.356775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:11.356775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:11.356775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:11.356775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:11.356775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:11.356775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:11.356775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:11.356775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:11.356775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:11.356775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:11.356775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:11.356775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:11.356775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:11.358828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:11.358828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:11.358828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:11.358828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:11.358828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:11.358828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:11.358828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:11.358828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:11.358828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:11.358828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:11.358828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:11.358828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:11.358828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:11.358828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:11.358828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:11.358828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:11.360836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:11.360836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:11.360836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:11.360836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:11.360836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:11.360836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:11.360836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:11.360836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:11.360836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:11.360836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:11.360836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 04:13:11.360836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:11.360836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 243/479] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:13:11 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:11.365120 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db25627-c83a-4563-8549-dfd01c554e64 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:11.486314 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:11.497223 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db25627-c83a-4563-8549-dfd01c554e64 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:11.539476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4184deb4-f606-4976-af27-f99ea556fbcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:11.540097 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4184deb4-f606-4976-af27-f99ea556fbcd None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:11.540375 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4184deb4-f606-4976-af27-f99ea556fbcd None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:11.540647 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4184deb4-f606-4976-af27-f99ea556fbcd None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:11.541394 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4184deb4-f606-4976-af27-f99ea556fbcd None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:11.541699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4184deb4-f606-4976-af27-f99ea556fbcd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:11.543083 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4184deb4-f606-4976-af27-f99ea556fbcd None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:11.551471 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 237/480] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:11 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:12.675124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812a67ff-49e7-488d-984c-c9d10dfdc7e6 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:12.675843 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812a67ff-49e7-488d-984c-c9d10dfdc7e6 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:12.675843 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812a67ff-49e7-488d-984c-c9d10dfdc7e6 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:12.677315 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 244/481] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:12.682809 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e25ab-bb44-41ec-a2c9-8ff698f48e48 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:12.683331 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e25ab-bb44-41ec-a2c9-8ff698f48e48 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:12.683331 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e25ab-bb44-41ec-a2c9-8ff698f48e48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:12.706079 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ad6e25ab-bb44-41ec-a2c9-8ff698f48e48 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:12.707301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ad6e25ab-bb44-41ec-a2c9-8ff698f48e48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:12.807314 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 238/482] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:12.934640 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e700bc4-6481-4511-bf92-a2f6195a4636 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:12.934939 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e700bc4-6481-4511-bf92-a2f6195a4636 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:12.935187 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e700bc4-6481-4511-bf92-a2f6195a4636 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:12.936804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 245/483] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:12.941676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5e95e7-c489-4d8f-8971-4cd2caba3fdc None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:12.941962 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5e95e7-c489-4d8f-8971-4cd2caba3fdc None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:12.942235 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5e95e7-c489-4d8f-8971-4cd2caba3fdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:12.964488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ca5e95e7-c489-4d8f-8971-4cd2caba3fdc None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:12.965696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ca5e95e7-c489-4d8f-8971-4cd2caba3fdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:13.055189 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 239/484] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:13.248547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e700bc4-6481-4511-bf92-a2f6195a4636 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:13.253671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ea3851-8ed6-4c0f-bda6-8e81531f162b None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:13.253957 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ea3851-8ed6-4c0f-bda6-8e81531f162b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:13.254230 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ea3851-8ed6-4c0f-bda6-8e81531f162b None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:13.256694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 240/485] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:13 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:13.265081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51547199-0934-489f-b661-b2a1df6929eb None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:13.265081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51547199-0934-489f-b661-b2a1df6929eb None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:13.265081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51547199-0934-489f-b661-b2a1df6929eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:13.295551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-51547199-0934-489f-b661-b2a1df6929eb None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:13.299683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-51547199-0934-489f-b661-b2a1df6929eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:13.313420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:13.406292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e700bc4-6481-4511-bf92-a2f6195a4636 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:13.417166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 241/486] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:13.418257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51547199-0934-489f-b661-b2a1df6929eb None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:13.454667 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7b060f-ef90-46b7-bb99-6904e3575006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:13.455406 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7b060f-ef90-46b7-bb99-6904e3575006 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:13.455692 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7b060f-ef90-46b7-bb99-6904e3575006 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:13.455963 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7b060f-ef90-46b7-bb99-6904e3575006 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:13.469227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7b060f-ef90-46b7-bb99-6904e3575006 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:13.475198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-0f7b060f-ef90-46b7-bb99-6904e3575006 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:13.475198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:13.475198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:13.475198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:13.475198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:13.475198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:13.475198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:13.475198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:13.475198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:13.475198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:13.475198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:13.475198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:13.475198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:13.475198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:13.475198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:13.475198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:13.476254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:13.476254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:13.476254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:13.476254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:13.476254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:13.476254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:13.476254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:13.476254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:13.476254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:13.476254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:13.476254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:13.476254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:13.476254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:13.476254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:13.476254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:13.476254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:13.477257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:13.477257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:13.477257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:13.477257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:13.477257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:13.477257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:13.477257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:13.477257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:13.477257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:13.477257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:13.477257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:13.477257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:13.480070 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 246/487] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:13.481632 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbf6aa0-3ed8-4edf-b183-7c1985148ce9 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:13.482432 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbf6aa0-3ed8-4edf-b183-7c1985148ce9 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:13.482852 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbf6aa0-3ed8-4edf-b183-7c1985148ce9 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:13.484597 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 247/488] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:13.485511 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbf6aa0-3ed8-4edf-b183-7c1985148ce9 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:13.517909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:13.549207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51547199-0934-489f-b661-b2a1df6929eb None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:13.587306 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86cc1210-6ad1-4308-9144-677a3b293d3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:13.587949 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cc1210-6ad1-4308-9144-677a3b293d3d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:13.588225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cc1210-6ad1-4308-9144-677a3b293d3d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:13.588498 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cc1210-6ad1-4308-9144-677a3b293d3d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:13.600680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86cc1210-6ad1-4308-9144-677a3b293d3d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:13.606353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-86cc1210-6ad1-4308-9144-677a3b293d3d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:13.606353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:13.606353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:13.606353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:13.606353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:13.606353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:13.606353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:13.606353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:13.606353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:13.606353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:13.606353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:13.606353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:13.606353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:13.606353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:13.606353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:13.606353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:13.607268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:13.607268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:13.607268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:13.607268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:13.607268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:13.607268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:13.607268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:13.607268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:13.607268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:13.607268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:13.607268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:13.607268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:13.607268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:13.607268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:13.607268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:13.607268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:13.608635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:13.608635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:13.608635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:13.608635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:13.608635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:13.608635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:13.608635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:13.608635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:13.608635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:13.608635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:13.608635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:13.608635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:13.608635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 242/489] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:13.609361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80de4852-f063-457f-a510-5c1a9d1813b0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:13.609361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80de4852-f063-457f-a510-5c1a9d1813b0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:13.609737 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80de4852-f063-457f-a510-5c1a9d1813b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:13.611466 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbf6aa0-3ed8-4edf-b183-7c1985148ce9 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:13.616782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:13.630511 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-80de4852-f063-457f-a510-5c1a9d1813b0 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:13.632057 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-80de4852-f063-457f-a510-5c1a9d1813b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:13.660501 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7d3df6-384d-4c73-a235-77d6f834edc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:13.662308 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7d3df6-384d-4c73-a235-77d6f834edc6 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:13.662839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7d3df6-384d-4c73-a235-77d6f834edc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:13.663581 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7d3df6-384d-4c73-a235-77d6f834edc6 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:13.664889 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7d3df6-384d-4c73-a235-77d6f834edc6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:13.665421 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7d3df6-384d-4c73-a235-77d6f834edc6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:13.667131 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7d3df6-384d-4c73-a235-77d6f834edc6 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:13.678150 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 248/490] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:13:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:13.678728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7d3df6-384d-4c73-a235-77d6f834edc6 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:13.711853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 243/491] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:13.712717 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80de4852-f063-457f-a510-5c1a9d1813b0 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:13.771304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:13.784682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7d3df6-384d-4c73-a235-77d6f834edc6 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:13.821980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2c11aa-389e-44e2-bcf2-3d3f7c18d8ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:13.822593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2c11aa-389e-44e2-bcf2-3d3f7c18d8ee None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:13.822868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2c11aa-389e-44e2-bcf2-3d3f7c18d8ee None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:13.823138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2c11aa-389e-44e2-bcf2-3d3f7c18d8ee None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:13.823813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2c11aa-389e-44e2-bcf2-3d3f7c18d8ee None admin] RBAC: Adding query filter params (name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:13.824112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2c11aa-389e-44e2-bcf2-3d3f7c18d8ee None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:13.824801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80de4852-f063-457f-a510-5c1a9d1813b0 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:13.825507 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2c11aa-389e-44e2-bcf2-3d3f7c18d8ee None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:13.838739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 249/492] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:13:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:13.839730 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2c11aa-389e-44e2-bcf2-3d3f7c18d8ee None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:13.856334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:13.863105 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906ea176-9934-4ac7-b59f-ee0052f77ee5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:13.863746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906ea176-9934-4ac7-b59f-ee0052f77ee5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:13.864078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906ea176-9934-4ac7-b59f-ee0052f77ee5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:13.864435 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906ea176-9934-4ac7-b59f-ee0052f77ee5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:13.878232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906ea176-9934-4ac7-b59f-ee0052f77ee5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:13.880183 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906ea176-9934-4ac7-b59f-ee0052f77ee5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:13.895810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-906ea176-9934-4ac7-b59f-ee0052f77ee5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:13.895810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:13.895810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:13.895810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:13.895810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:13.895810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:13.895810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:13.895810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:13.895810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:13.895810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:13.895810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:13.895810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:13.895810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:13.895810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:13.895810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:13.895810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:13.896720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:13.896720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:13.896720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:13.896720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:13.896720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:13.896720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:13.896720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:13.896720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:13.896720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:13.896720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:13.896720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:13.896720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:13.896720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:13.896720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:13.896720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:13.896720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:13.896720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:13.898593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:13.898593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:13.898593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:13.898593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 244/493] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:13.898593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906ea176-9934-4ac7-b59f-ee0052f77ee5 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:13.950000 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2c11aa-389e-44e2-bcf2-3d3f7c18d8ee None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:13.986085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3613e7c-ed60-4dc6-8a92-59992febd61d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:13.986718 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3613e7c-ed60-4dc6-8a92-59992febd61d None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:13.986998 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3613e7c-ed60-4dc6-8a92-59992febd61d None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:13.987271 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3613e7c-ed60-4dc6-8a92-59992febd61d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:13.998979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3613e7c-ed60-4dc6-8a92-59992febd61d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:14.004717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-a3613e7c-ed60-4dc6-8a92-59992febd61d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:14.004717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:14.004717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:14.004717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:14.004717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:14.004717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:14.004717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:14.004717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:14.004717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:14.004717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:14.004717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:14.004717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:14.004717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:14.004717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:14.004717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:14.004717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:14.007089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:14.007089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:14.007089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:14.007089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:14.007089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:14.007089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:14.007089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:14.007089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:14.007089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:14.007089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:14.007089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:14.007089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:14.007089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:14.007089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:14.007089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:14.007089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:14.009183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:14.009183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:14.009183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:14.009183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:14.009183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:14.009183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:14.009183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:14.009183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:14.009183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:14.009183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:14.009183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:14.009183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:14.009183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 250/494] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:14.009183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3613e7c-ed60-4dc6-8a92-59992febd61d None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:14.011316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906ea176-9934-4ac7-b59f-ee0052f77ee5 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:14.027325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:14.045853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b66505-1419-4d89-873e-851c17ac67f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:14.046485 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b66505-1419-4d89-873e-851c17ac67f3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:14.046767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b66505-1419-4d89-873e-851c17ac67f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:14.047044 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b66505-1419-4d89-873e-851c17ac67f3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:14.058118 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b66505-1419-4d89-873e-851c17ac67f3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:14.059584 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b66505-1419-4d89-873e-851c17ac67f3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:14.074014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-33b66505-1419-4d89-873e-851c17ac67f3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:14.074014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:14.074014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:14.074014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:14.074014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:14.074014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:14.074014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:14.074014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:14.074014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:14.074014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:14.074014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:14.074014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:14.074014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:14.074014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:14.074014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:14.074014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:14.074930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:14.074930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:14.074930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:14.074930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:14.074930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:14.074930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:14.074930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:14.074930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:14.074930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:14.074930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:14.074930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:14.074930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:14.074930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:14.074930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:14.074930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:14.074930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:14.074930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:14.077274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:14.077274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:14.077274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:14.077274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 245/495] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:14.077274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b66505-1419-4d89-873e-851c17ac67f3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:14.113649 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:14.139449 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3613e7c-ed60-4dc6-8a92-59992febd61d None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:14.176359 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c134ea20-bb95-45a1-8244-15406a317e4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:14.176993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c134ea20-bb95-45a1-8244-15406a317e4f None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:14.177298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c134ea20-bb95-45a1-8244-15406a317e4f None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:14.177569 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c134ea20-bb95-45a1-8244-15406a317e4f None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:14.178214 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c134ea20-bb95-45a1-8244-15406a317e4f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:14.178515 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c134ea20-bb95-45a1-8244-15406a317e4f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:14.180261 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c134ea20-bb95-45a1-8244-15406a317e4f None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:14.191446 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b66505-1419-4d89-873e-851c17ac67f3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:14.192067 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 251/496] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:14.192923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c134ea20-bb95-45a1-8244-15406a317e4f None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:14.226621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464e3693-f3fa-42c3-a0a1-7259b23c05eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:14.227224 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464e3693-f3fa-42c3-a0a1-7259b23c05eb None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:14.227499 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464e3693-f3fa-42c3-a0a1-7259b23c05eb None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:14.227770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464e3693-f3fa-42c3-a0a1-7259b23c05eb None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:14.228431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464e3693-f3fa-42c3-a0a1-7259b23c05eb None admin] RBAC: Adding query filter params (name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:14.228729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464e3693-f3fa-42c3-a0a1-7259b23c05eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:14.230128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464e3693-f3fa-42c3-a0a1-7259b23c05eb None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:14.238260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 246/497] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:13:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:14.238872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464e3693-f3fa-42c3-a0a1-7259b23c05eb None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:14.266740 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:14.298146 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c134ea20-bb95-45a1-8244-15406a317e4f None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:14.332719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a87072-c258-4d8f-926b-f82e8e76a542 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:14.333370 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a87072-c258-4d8f-926b-f82e8e76a542 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:14.333652 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a87072-c258-4d8f-926b-f82e8e76a542 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:14.333918 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a87072-c258-4d8f-926b-f82e8e76a542 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:14.334869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:14.339140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464e3693-f3fa-42c3-a0a1-7259b23c05eb None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:14.344264 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a87072-c258-4d8f-926b-f82e8e76a542 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:14.349113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-95a87072-c258-4d8f-926b-f82e8e76a542 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:14.349113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:14.349113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:14.349113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:14.349113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:14.349113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:14.349113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:14.349113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:14.349113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:14.349113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:14.349113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:14.349113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:14.349113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:14.349113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:14.349113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:14.349113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:14.350026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:14.350026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:14.350026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:14.350026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:14.350026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:14.350026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:14.350026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:14.350026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:14.350026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:14.350026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:14.350026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:14.350026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:14.350026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:14.350026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:14.350026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:14.350026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:14.350829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:14.350829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:14.350829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:14.350829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:14.350829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:14.350829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:14.350829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:14.350829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:14.350829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:14.350829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:14.350829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:14.350829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:14.350829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 252/498] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:14.351472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886e1f0a-0c1d-4b64-b79c-25bc87449a94 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:14.351916 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886e1f0a-0c1d-4b64-b79c-25bc87449a94 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:14.351916 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886e1f0a-0c1d-4b64-b79c-25bc87449a94 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:14.353825 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 253/499] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:14.354526 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886e1f0a-0c1d-4b64-b79c-25bc87449a94 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:14.379234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007dfed8-4b77-4538-b506-462c3b9f8044 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:14.380162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007dfed8-4b77-4538-b506-462c3b9f8044 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:14.380448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007dfed8-4b77-4538-b506-462c3b9f8044 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:14.380720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007dfed8-4b77-4538-b506-462c3b9f8044 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:14.381642 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007dfed8-4b77-4538-b506-462c3b9f8044 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:14.382228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007dfed8-4b77-4538-b506-462c3b9f8044 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:14.383317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007dfed8-4b77-4538-b506-462c3b9f8044 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:14.394708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 247/500] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:14.395498 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007dfed8-4b77-4538-b506-462c3b9f8044 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:14.462853 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886e1f0a-0c1d-4b64-b79c-25bc87449a94 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:14.493352 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:14.495619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007dfed8-4b77-4538-b506-462c3b9f8044 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:14.499085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75449312-5ed1-4c8b-8bc7-39ddc8c57409 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:14.499711 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75449312-5ed1-4c8b-8bc7-39ddc8c57409 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:14.499990 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75449312-5ed1-4c8b-8bc7-39ddc8c57409 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:14.500257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75449312-5ed1-4c8b-8bc7-39ddc8c57409 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:14.508015 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75449312-5ed1-4c8b-8bc7-39ddc8c57409 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:14.509489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75449312-5ed1-4c8b-8bc7-39ddc8c57409 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:14.522813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-75449312-5ed1-4c8b-8bc7-39ddc8c57409 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 30 04:13:14.522813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:14.522813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:14.522813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:14.522813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:14.522813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:14.522813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:14.522813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:14.522813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:14.522813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:14.522813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:14.522813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:14.522813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:14.522813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:14.522813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:14.522813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:14.523770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:14.523770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:14.523770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:14.523770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:14.523770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:14.523770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:14.523770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:14.523770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:14.523770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:14.523770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:14.523770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:14.523770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:14.523770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:14.523770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:14.523770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:14.523770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:14.523770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:14.523770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:14.524666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:14.524666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:14.524666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:14.524666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:14.524666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:14.524666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:14.524666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:14.524666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:14.524666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 30 04:13:14.524666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:14.524666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 254/501] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:13:14 2022] GET /identity/v3/users/glance => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:14.524666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75449312-5ed1-4c8b-8bc7-39ddc8c57409 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:14.535841 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a81612a-a8a5-43d3-ac0f-83b1ab2155bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:14.536513 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a81612a-a8a5-43d3-ac0f-83b1ab2155bf None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:14.536794 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a81612a-a8a5-43d3-ac0f-83b1ab2155bf None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:14.537093 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a81612a-a8a5-43d3-ac0f-83b1ab2155bf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:14.547136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a81612a-a8a5-43d3-ac0f-83b1ab2155bf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:14.548657 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a81612a-a8a5-43d3-ac0f-83b1ab2155bf None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:14.562956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-1a81612a-a8a5-43d3-ac0f-83b1ab2155bf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:14.562956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:14.562956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:14.562956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:14.562956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:14.562956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:14.562956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:14.562956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:14.562956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:14.562956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:14.562956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:14.562956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:14.562956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:14.562956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:14.562956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:14.562956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:14.564032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:14.564032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:14.564032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:14.564032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:14.564032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:14.564032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:14.564032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:14.564032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:14.564032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:14.564032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:14.564032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:14.564032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:14.564032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:14.564032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:14.564032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:14.564032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:14.564032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:14.564946 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:14.564946 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:14.564946 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:14.564946 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 248/502] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:14.565204 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82192b7-e755-462b-995d-a217ce82bc79 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:14.565495 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82192b7-e755-462b-995d-a217ce82bc79 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:14.565771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82192b7-e755-462b-995d-a217ce82bc79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:14.584186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-f82192b7-e755-462b-995d-a217ce82bc79 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:14.585419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f82192b7-e755-462b-995d-a217ce82bc79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:14.588131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:14.628707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75449312-5ed1-4c8b-8bc7-39ddc8c57409 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:14.660607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 249/503] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:14.661903 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82192b7-e755-462b-995d-a217ce82bc79 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:14.677194 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b255662-2cb1-4b54-8d4a-fe0309aa62b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:14.678013 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b255662-2cb1-4b54-8d4a-fe0309aa62b2 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:14.678298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b255662-2cb1-4b54-8d4a-fe0309aa62b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:14.678891 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b255662-2cb1-4b54-8d4a-fe0309aa62b2 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:14.679239 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b255662-2cb1-4b54-8d4a-fe0309aa62b2 None admin] RBAC: Adding query filter params (name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:14.679542 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b255662-2cb1-4b54-8d4a-fe0309aa62b2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:14.680956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b255662-2cb1-4b54-8d4a-fe0309aa62b2 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:14.694167 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 255/504] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:13:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:14.694167 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b255662-2cb1-4b54-8d4a-fe0309aa62b2 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:14.749392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:14.772747 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82192b7-e755-462b-995d-a217ce82bc79 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:14.800878 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b255662-2cb1-4b54-8d4a-fe0309aa62b2 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:14.809566 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2d06cd-cac4-4105-82ec-44fe02d7796d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:14.810397 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2d06cd-cac4-4105-82ec-44fe02d7796d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:14.810819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2d06cd-cac4-4105-82ec-44fe02d7796d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:14.811245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2d06cd-cac4-4105-82ec-44fe02d7796d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:14.819063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2d06cd-cac4-4105-82ec-44fe02d7796d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:14.820924 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2d06cd-cac4-4105-82ec-44fe02d7796d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:14.824744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:14.837153 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4afd9a0-6aee-4c8b-8731-a95afb952cbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:14.837321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-8c2d06cd-cac4-4105-82ec-44fe02d7796d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:13:14.837321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:14.837321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:14.837321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:14.837321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:14.837321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:14.837321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:14.837321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:14.837321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:14.837321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:14.837321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:14.837321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:14.837321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:14.837321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:14.837321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:14.837321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:14.838167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:14.838167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:14.838167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:14.838167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:14.838167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:14.838167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:14.838167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:14.838167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:14.838167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:14.838167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:14.838167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:14.838167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:14.838167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:14.838167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:14.838167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:14.838167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:14.838167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:14.838167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:14.839035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:14.839035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:14.839035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:14.839035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:14.839035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:14.839035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:14.839035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:14.839035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:14.839035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:13:14.839035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:14.839537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4afd9a0-6aee-4c8b-8731-a95afb952cbb None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:14.839537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4afd9a0-6aee-4c8b-8731-a95afb952cbb None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:14.839537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4afd9a0-6aee-4c8b-8731-a95afb952cbb None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:14.839537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4afd9a0-6aee-4c8b-8731-a95afb952cbb None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:14.839537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4afd9a0-6aee-4c8b-8731-a95afb952cbb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:14.840836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4afd9a0-6aee-4c8b-8731-a95afb952cbb None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:14.842998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 250/505] 10.222.0.85 () {56 vars in 1093 bytes} [Tue Aug 30 04:13:14 2022] GET /identity/v3/users/nova => generated 82 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:14.843845 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2d06cd-cac4-4105-82ec-44fe02d7796d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:14.887598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 256/506] 10.222.0.85 () {56 vars in 1137 bytes} [Tue Aug 30 04:13:14 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:14.888264 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4afd9a0-6aee-4c8b-8731-a95afb952cbb None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:14.953982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2d06cd-cac4-4105-82ec-44fe02d7796d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:14.988115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5556081-9d1f-4cbf-a496-f6ae40e45e72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:14.988725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5556081-9d1f-4cbf-a496-f6ae40e45e72 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:14.989003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5556081-9d1f-4cbf-a496-f6ae40e45e72 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:14.989303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5556081-9d1f-4cbf-a496-f6ae40e45e72 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:14.990453 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5556081-9d1f-4cbf-a496-f6ae40e45e72 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:14.990704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4afd9a0-6aee-4c8b-8731-a95afb952cbb None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:14.990803 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5556081-9d1f-4cbf-a496-f6ae40e45e72 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:14.992133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5556081-9d1f-4cbf-a496-f6ae40e45e72 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:15.001203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:15.003721 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 251/507] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:15.004528 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5556081-9d1f-4cbf-a496-f6ae40e45e72 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:15.026589 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5920f8-0bd6-4c90-8ec7-addad43b3b3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:15.027211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5920f8-0bd6-4c90-8ec7-addad43b3b3b None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:15.027495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5920f8-0bd6-4c90-8ec7-addad43b3b3b None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:15.027772 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5920f8-0bd6-4c90-8ec7-addad43b3b3b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:15.037614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5920f8-0bd6-4c90-8ec7-addad43b3b3b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:15.039114 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5920f8-0bd6-4c90-8ec7-addad43b3b3b None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:15.054039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-3a5920f8-0bd6-4c90-8ec7-addad43b3b3b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:15.054039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:15.054039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:15.054039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:15.054039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:15.054039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:15.054039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:15.054039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:15.054039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:15.054039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:15.054039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:15.054039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:15.054039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:15.054039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:15.054039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:15.054039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:15.055111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:15.055111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:15.055111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:15.055111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:15.055111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:15.055111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:15.055111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:15.055111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:15.055111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:15.055111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:15.055111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:15.055111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:15.055111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:15.055111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:15.055111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:15.055111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:15.055111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:15.056337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:15.056337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:15.056337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:15.056337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 257/508] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:15.056337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5920f8-0bd6-4c90-8ec7-addad43b3b3b None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:15.068248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:15.111230 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5556081-9d1f-4cbf-a496-f6ae40e45e72 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:15.146497 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93b5ded-c8e6-4bc7-a521-03dfad5c1c7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:15.147113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93b5ded-c8e6-4bc7-a521-03dfad5c1c7a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:15.147394 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93b5ded-c8e6-4bc7-a521-03dfad5c1c7a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:15.147668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93b5ded-c8e6-4bc7-a521-03dfad5c1c7a None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:15.148394 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93b5ded-c8e6-4bc7-a521-03dfad5c1c7a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:15.148701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93b5ded-c8e6-4bc7-a521-03dfad5c1c7a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:15.150207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93b5ded-c8e6-4bc7-a521-03dfad5c1c7a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:15.158907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5920f8-0bd6-4c90-8ec7-addad43b3b3b None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:15.186363 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 252/509] 10.222.0.85 () {56 vars in 1133 bytes} [Tue Aug 30 04:13:15 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:15.187076 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93b5ded-c8e6-4bc7-a521-03dfad5c1c7a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:15.194307 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57500dca-a619-4cd2-8c36-c67ea99c2167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:15.194917 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57500dca-a619-4cd2-8c36-c67ea99c2167 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:15.195199 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57500dca-a619-4cd2-8c36-c67ea99c2167 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:15.195474 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57500dca-a619-4cd2-8c36-c67ea99c2167 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:15.225405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:15.263922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57500dca-a619-4cd2-8c36-c67ea99c2167 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:15.265527 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57500dca-a619-4cd2-8c36-c67ea99c2167 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:15.292856 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93b5ded-c8e6-4bc7-a521-03dfad5c1c7a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:15.311649 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:15.328441 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f97e35-ac64-40eb-8ca0-fed0bfcac797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:15.329097 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f97e35-ac64-40eb-8ca0-fed0bfcac797 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:15.329385 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f97e35-ac64-40eb-8ca0-fed0bfcac797 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:15.329653 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f97e35-ac64-40eb-8ca0-fed0bfcac797 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:15.339527 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f97e35-ac64-40eb-8ca0-fed0bfcac797 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:15.340993 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f97e35-ac64-40eb-8ca0-fed0bfcac797 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:15.355969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-12f97e35-ac64-40eb-8ca0-fed0bfcac797 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:15.355969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:15.355969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:15.355969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:15.355969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:15.355969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:15.355969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:15.355969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:15.355969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:15.355969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:15.355969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:15.355969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:15.355969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:15.355969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:15.355969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:15.355969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:15.356895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:15.356895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:15.356895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:15.356895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:15.356895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:15.356895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:15.356895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:15.356895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:15.356895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:15.356895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:15.356895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:15.356895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:15.356895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:15.356895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:15.356895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:15.356895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:15.356895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:15.357781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:15.357781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:15.357781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:15.357781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 253/510] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:15.358003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f97e35-ac64-40eb-8ca0-fed0bfcac797 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:15.368724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 258/511] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:13:15.369606 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57500dca-a619-4cd2-8c36-c67ea99c2167 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:15.428984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:15.456251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f97e35-ac64-40eb-8ca0-fed0bfcac797 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:15.469134 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57500dca-a619-4cd2-8c36-c67ea99c2167 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:15.489964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e0fcb7-7b22-4ac9-ba69-0876a4c03bb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:15.490588 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e0fcb7-7b22-4ac9-ba69-0876a4c03bb5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:15.490869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e0fcb7-7b22-4ac9-ba69-0876a4c03bb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:15.491135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e0fcb7-7b22-4ac9-ba69-0876a4c03bb5 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:15.498475 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e0fcb7-7b22-4ac9-ba69-0876a4c03bb5 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:15.499923 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e0fcb7-7b22-4ac9-ba69-0876a4c03bb5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:15.502122 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449bb047-2303-4ce1-b3d9-6f90f109cbcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:15.503169 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449bb047-2303-4ce1-b3d9-6f90f109cbcd None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:15.503447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449bb047-2303-4ce1-b3d9-6f90f109cbcd None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:15.503718 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449bb047-2303-4ce1-b3d9-6f90f109cbcd None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:15.504352 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449bb047-2303-4ce1-b3d9-6f90f109cbcd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:15.504651 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449bb047-2303-4ce1-b3d9-6f90f109cbcd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:15.506884 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449bb047-2303-4ce1-b3d9-6f90f109cbcd None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:15.511683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-c0e0fcb7-7b22-4ac9-ba69-0876a4c03bb5 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 04:13:15.511683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:15.511683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:15.511683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:15.511683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:15.511683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:15.511683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:15.511683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:15.511683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:15.511683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:15.511683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:15.511683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:15.511683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:15.511683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:15.511683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:15.511683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:15.512544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:15.512544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:15.512544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:15.512544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:15.512544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:15.512544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:15.512544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:15.512544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:15.512544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:15.512544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:15.512544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:15.512544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:15.512544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:15.512544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:15.512544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:15.512544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:15.512544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:15.512544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:15.513472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:15.513472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:15.513472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:15.513472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:15.513472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:15.513472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:15.513472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:15.513472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:15.513472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 04:13:15.513472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:15.513472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 254/512] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:13:15 2022] GET /identity/v3/users/cinder => generated 84 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:15.514032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e0fcb7-7b22-4ac9-ba69-0876a4c03bb5 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:15.522362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 259/513] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:15.523163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449bb047-2303-4ce1-b3d9-6f90f109cbcd None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:15.549918 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:15.618877 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e0fcb7-7b22-4ac9-ba69-0876a4c03bb5 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:15.622030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449bb047-2303-4ce1-b3d9-6f90f109cbcd None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:15.644042 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:15.651909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad34608-00c8-4f51-a482-f4ab1c7eb1eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:15.652513 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad34608-00c8-4f51-a482-f4ab1c7eb1eb None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:15.652788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad34608-00c8-4f51-a482-f4ab1c7eb1eb None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:15.653088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad34608-00c8-4f51-a482-f4ab1c7eb1eb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:15.656101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5987ff32-10e2-474c-8112-6ad28f0d2814 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:15.656704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5987ff32-10e2-474c-8112-6ad28f0d2814 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:15.656980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5987ff32-10e2-474c-8112-6ad28f0d2814 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:15.657296 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5987ff32-10e2-474c-8112-6ad28f0d2814 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:15.657929 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5987ff32-10e2-474c-8112-6ad28f0d2814 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:15.658228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5987ff32-10e2-474c-8112-6ad28f0d2814 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:15.659590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5987ff32-10e2-474c-8112-6ad28f0d2814 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:15.671841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 260/514] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:15.672628 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5987ff32-10e2-474c-8112-6ad28f0d2814 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:15.719248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad34608-00c8-4f51-a482-f4ab1c7eb1eb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:15.720727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad34608-00c8-4f51-a482-f4ab1c7eb1eb None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:15.750417 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:15.774422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5987ff32-10e2-474c-8112-6ad28f0d2814 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:15.807458 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66337932-95bf-4fbb-a151-9a2e6d4e3dde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:15.808086 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66337932-95bf-4fbb-a151-9a2e6d4e3dde None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:15.808371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66337932-95bf-4fbb-a151-9a2e6d4e3dde None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:15.808642 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66337932-95bf-4fbb-a151-9a2e6d4e3dde None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:15.819021 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66337932-95bf-4fbb-a151-9a2e6d4e3dde None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:15.820462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66337932-95bf-4fbb-a151-9a2e6d4e3dde None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:15.821015 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 255/515] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:13:15.821015 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad34608-00c8-4f51-a482-f4ab1c7eb1eb None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:15.835728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-66337932-95bf-4fbb-a151-9a2e6d4e3dde None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:15.835728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:15.835728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:15.835728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:15.835728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:15.835728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:15.835728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:15.835728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:15.835728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:15.835728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:15.835728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:15.835728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:15.835728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:15.835728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:15.835728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:15.835728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:15.836814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:15.836814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:15.836814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:15.836814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:15.836814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:15.836814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:15.836814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:15.836814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:15.836814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:15.836814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:15.836814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:15.836814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:15.836814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:15.836814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:15.836814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:15.836814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:15.836814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:15.838113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:15.838113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:15.838113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:15.838113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 261/516] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:15.838113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66337932-95bf-4fbb-a151-9a2e6d4e3dde None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:15.875283 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:15.930299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad34608-00c8-4f51-a482-f4ab1c7eb1eb None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:15.939338 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66337932-95bf-4fbb-a151-9a2e6d4e3dde None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:15.959092 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:15.965127 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5491096-4b0c-4da5-8430-85044014f0fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:15.965730 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5491096-4b0c-4da5-8430-85044014f0fe None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:15.966004 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5491096-4b0c-4da5-8430-85044014f0fe None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:15.966276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5491096-4b0c-4da5-8430-85044014f0fe None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:15.966992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5491096-4b0c-4da5-8430-85044014f0fe None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:15.967292 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5491096-4b0c-4da5-8430-85044014f0fe None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:15.968736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5491096-4b0c-4da5-8430-85044014f0fe None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:15.973032 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed64d3e5-2ca9-4eab-9d0d-1c5887c71c99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:15.973670 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed64d3e5-2ca9-4eab-9d0d-1c5887c71c99 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:15.973983 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed64d3e5-2ca9-4eab-9d0d-1c5887c71c99 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:15.974255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed64d3e5-2ca9-4eab-9d0d-1c5887c71c99 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:15.981674 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed64d3e5-2ca9-4eab-9d0d-1c5887c71c99 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:15.983113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed64d3e5-2ca9-4eab-9d0d-1c5887c71c99 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:15.986100 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 256/517] 10.222.0.85 () {56 vars in 1137 bytes} [Tue Aug 30 04:13:15 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:15.986873 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5491096-4b0c-4da5-8430-85044014f0fe None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:15.997155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-ed64d3e5-2ca9-4eab-9d0d-1c5887c71c99 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 30 04:13:15.997155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:15.997155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:15.997155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:15.997155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:15.997155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:15.997155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:15.997155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:15.997155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:15.997155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:15.997155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:15.997155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:15.997155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:15.997155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:15.997155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:15.997155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:15.998277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:15.998277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:15.998277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:15.998277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:15.998277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:15.998277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:15.998277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:15.998277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:15.998277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:15.998277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:15.998277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:15.998277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:15.998277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:15.998277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:15.998277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:15.998277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:15.998277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:15.998277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:15.999305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:15.999305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:15.999305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:15.999305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:15.999305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:15.999305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:15.999305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:15.999305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:15.999305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 30 04:13:15.999305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:15.999305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 262/518] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:15 2022] GET /identity/v3/users/neutron => generated 85 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:15.999305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed64d3e5-2ca9-4eab-9d0d-1c5887c71c99 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:16.087490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5491096-4b0c-4da5-8430-85044014f0fe None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:16.090682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:16.097973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed64d3e5-2ca9-4eab-9d0d-1c5887c71c99 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:16.123353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5dbfcd-1951-4cab-b745-c07c57976112 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:16.123961 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5dbfcd-1951-4cab-b745-c07c57976112 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:16.124237 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5dbfcd-1951-4cab-b745-c07c57976112 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:16.124506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5dbfcd-1951-4cab-b745-c07c57976112 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:16.125166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5dbfcd-1951-4cab-b745-c07c57976112 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:16.126672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5dbfcd-1951-4cab-b745-c07c57976112 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:16.134727 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd579cb9-00f2-494c-896a-b97d5f03fc88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:16.135345 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd579cb9-00f2-494c-896a-b97d5f03fc88 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:16.135622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd579cb9-00f2-494c-896a-b97d5f03fc88 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:16.135888 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd579cb9-00f2-494c-896a-b97d5f03fc88 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:16.147444 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 257/519] 10.222.0.85 () {60 vars in 1132 bytes} [Tue Aug 30 04:13:15 2022] POST /identity/v3/users => generated 247 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:13:16.147711 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd579cb9-00f2-494c-896a-b97d5f03fc88 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:16.148377 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5dbfcd-1951-4cab-b745-c07c57976112 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:16.149271 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd579cb9-00f2-494c-896a-b97d5f03fc88 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:16.175266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-fd579cb9-00f2-494c-896a-b97d5f03fc88 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:16.175266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:16.175266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:16.175266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:16.175266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:16.175266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:16.175266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:16.175266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:16.175266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:16.175266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:16.175266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:16.175266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:16.175266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:16.175266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:16.175266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:16.175266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:16.176461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:16.176461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:16.176461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:16.176461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:16.176461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:16.176461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:16.176461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:16.176461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:16.176461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:16.176461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:16.176461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:16.176461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:16.176461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:16.176461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:16.176461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:16.176461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:16.176461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:16.178174 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:16.178174 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:16.178174 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:16.178649 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 263/520] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:16.179737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd579cb9-00f2-494c-896a-b97d5f03fc88 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:16.208321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:16.276505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5dbfcd-1951-4cab-b745-c07c57976112 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:16.292314 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd579cb9-00f2-494c-896a-b97d5f03fc88 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:16.312525 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70cccd52-66d2-474a-9319-b3482ef87454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:16.313179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cccd52-66d2-474a-9319-b3482ef87454 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:16.313459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cccd52-66d2-474a-9319-b3482ef87454 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:16.313737 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cccd52-66d2-474a-9319-b3482ef87454 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:16.314385 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70cccd52-66d2-474a-9319-b3482ef87454 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:16.314689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70cccd52-66d2-474a-9319-b3482ef87454 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:16.316068 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70cccd52-66d2-474a-9319-b3482ef87454 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:16.331652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 258/521] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:16.332699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70cccd52-66d2-474a-9319-b3482ef87454 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:16.335071 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0124b2e9-750b-4034-b466-fc2a966c41ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:16.335698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0124b2e9-750b-4034-b466-fc2a966c41ad None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:16.336109 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0124b2e9-750b-4034-b466-fc2a966c41ad None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:16.336388 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0124b2e9-750b-4034-b466-fc2a966c41ad None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:16.358163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:16.420417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0124b2e9-750b-4034-b466-fc2a966c41ad None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:16.421829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0124b2e9-750b-4034-b466-fc2a966c41ad None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:16.443055 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70cccd52-66d2-474a-9319-b3482ef87454 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:16.446364 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:16.484882 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9adf16-db43-49d9-9d34-068673a0c279 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:16.485881 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9adf16-db43-49d9-9d34-068673a0c279 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:16.486325 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9adf16-db43-49d9-9d34-068673a0c279 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:16.486783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9adf16-db43-49d9-9d34-068673a0c279 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:16.487932 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9adf16-db43-49d9-9d34-068673a0c279 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:16.488439 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9adf16-db43-49d9-9d34-068673a0c279 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:16.491523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9adf16-db43-49d9-9d34-068673a0c279 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:16.519762 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 259/522] 10.222.0.85 () {56 vars in 1139 bytes} [Tue Aug 30 04:13:16 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:16.519762 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9adf16-db43-49d9-9d34-068673a0c279 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:16.546364 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 264/523] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:13:16.547045 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0124b2e9-750b-4034-b466-fc2a966c41ad None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:16.597665 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:16.626321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9adf16-db43-49d9-9d34-068673a0c279 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:16.654961 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0124b2e9-750b-4034-b466-fc2a966c41ad None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:16.663748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2248e71-a81a-43ea-bc34-91c83262039f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:16.664359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2248e71-a81a-43ea-bc34-91c83262039f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:16.664649 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2248e71-a81a-43ea-bc34-91c83262039f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:16.664969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2248e71-a81a-43ea-bc34-91c83262039f None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:16.665641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2248e71-a81a-43ea-bc34-91c83262039f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:16.665941 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2248e71-a81a-43ea-bc34-91c83262039f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:16.667313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2248e71-a81a-43ea-bc34-91c83262039f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:16.679213 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 260/524] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:16.679949 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2248e71-a81a-43ea-bc34-91c83262039f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:16.699319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2095a366-599f-4808-9afb-6212190b1c62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:16.700231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2095a366-599f-4808-9afb-6212190b1c62 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:16.700691 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2095a366-599f-4808-9afb-6212190b1c62 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:16.701319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2095a366-599f-4808-9afb-6212190b1c62 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:16.706357 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:16.709180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2095a366-599f-4808-9afb-6212190b1c62 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:16.710783 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2095a366-599f-4808-9afb-6212190b1c62 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:16.723371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-2095a366-599f-4808-9afb-6212190b1c62 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:16.723371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:16.723371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:16.723371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:16.723371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:16.723371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:16.723371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:16.723371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:16.723371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:16.723371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:16.723371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:16.723371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:16.723371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:16.723371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:16.723371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:16.724439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:16.724439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:16.724439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:16.724439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:16.724439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:16.724439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:16.724439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:16.724439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:16.724439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:16.724439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:16.724439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:16.724439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:16.724439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:16.724439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:16.724439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:16.724439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:16.724439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:16.725464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:16.725464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:16.725464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:16.725464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:16.725464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:16.725464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:16.725464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:16.725464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:16.731814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 265/525] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:16.732837 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2095a366-599f-4808-9afb-6212190b1c62 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:16.791361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2248e71-a81a-43ea-bc34-91c83262039f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:16.828814 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bd3371-6e3d-48cc-8bf8-86c58a024f1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:16.829981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bd3371-6e3d-48cc-8bf8-86c58a024f1f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:16.830421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bd3371-6e3d-48cc-8bf8-86c58a024f1f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:16.830993 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bd3371-6e3d-48cc-8bf8-86c58a024f1f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:16.840214 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2095a366-599f-4808-9afb-6212190b1c62 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:16.848037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:16.878629 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd0f732-fcd7-4cbe-9052-258c547e4fbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:16.879381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd0f732-fcd7-4cbe-9052-258c547e4fbd None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:16.879661 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd0f732-fcd7-4cbe-9052-258c547e4fbd None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:16.880082 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd0f732-fcd7-4cbe-9052-258c547e4fbd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:16.891643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd0f732-fcd7-4cbe-9052-258c547e4fbd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:16.893252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd0f732-fcd7-4cbe-9052-258c547e4fbd None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:16.901285 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bd3371-6e3d-48cc-8bf8-86c58a024f1f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:16.902978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bd3371-6e3d-48cc-8bf8-86c58a024f1f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:16.908416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-cbd0f732-fcd7-4cbe-9052-258c547e4fbd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:16.908416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:16.908416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:16.908416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:16.908416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:16.908416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:16.908416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:16.908416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:16.908416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:16.908416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:16.908416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:16.908416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:16.908416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:16.908416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:16.908416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:16.908416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:16.909590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:16.909590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:16.909590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:16.909590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:16.909590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:16.909590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:16.909590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:16.909590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:16.909590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:16.909590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:16.909590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:16.909590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:16.909590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:16.909590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:16.909590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:16.909590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:16.909590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:16.910638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:16.910638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:16.910638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:16.913366 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 266/526] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:16.914205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd0f732-fcd7-4cbe-9052-258c547e4fbd None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:16.931561 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:17.018455 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd0f732-fcd7-4cbe-9052-258c547e4fbd None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:17.027565 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 261/527] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:13:17.027565 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bd3371-6e3d-48cc-8bf8-86c58a024f1f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:17.056142 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3575f63-197b-4a49-ac87-98cb2ece31b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:17.056867 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3575f63-197b-4a49-ac87-98cb2ece31b2 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:17.057181 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3575f63-197b-4a49-ac87-98cb2ece31b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:17.057460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3575f63-197b-4a49-ac87-98cb2ece31b2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:17.064226 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3575f63-197b-4a49-ac87-98cb2ece31b2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:17.065830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3575f63-197b-4a49-ac87-98cb2ece31b2 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:17.077460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-d3575f63-197b-4a49-ac87-98cb2ece31b2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:17.077460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:17.077460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:17.077460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:17.077460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:17.077460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:17.077460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:17.077460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:17.077460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:17.077460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:17.077460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:17.077460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:17.077460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:17.077460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:17.077460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:17.078417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:17.078417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:17.078417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:17.078417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:17.078417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:17.078417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:17.078417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:17.078417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:17.078417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:17.078417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:17.078417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:17.078417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:17.078417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:17.078417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:17.078417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:17.078417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:17.078417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:17.079278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:17.079278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:17.079278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:17.079278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:17.079278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:17.079278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:17.079278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:17.079278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:17.079278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 267/528] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:17.079735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3575f63-197b-4a49-ac87-98cb2ece31b2 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:17.104564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:17.141330 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bd3371-6e3d-48cc-8bf8-86c58a024f1f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:17.164831 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:17.179608 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9377686c-aa5f-484b-8be7-b7b99b60c42e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:17.180328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9377686c-aa5f-484b-8be7-b7b99b60c42e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:17.180766 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9377686c-aa5f-484b-8be7-b7b99b60c42e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:17.181030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9377686c-aa5f-484b-8be7-b7b99b60c42e None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:17.182032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9377686c-aa5f-484b-8be7-b7b99b60c42e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:17.182158 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9377686c-aa5f-484b-8be7-b7b99b60c42e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:17.183727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9377686c-aa5f-484b-8be7-b7b99b60c42e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:17.190829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3575f63-197b-4a49-ac87-98cb2ece31b2 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:17.202094 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 262/529] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:17.202094 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9377686c-aa5f-484b-8be7-b7b99b60c42e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:17.236163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22da195-ee9f-4d17-9b28-6303d1f2bb6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:17.237368 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22da195-ee9f-4d17-9b28-6303d1f2bb6e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:17.237968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22da195-ee9f-4d17-9b28-6303d1f2bb6e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:17.238547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22da195-ee9f-4d17-9b28-6303d1f2bb6e None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:17.239934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22da195-ee9f-4d17-9b28-6303d1f2bb6e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:17.240557 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22da195-ee9f-4d17-9b28-6303d1f2bb6e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:17.243271 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22da195-ee9f-4d17-9b28-6303d1f2bb6e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:17.263471 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 268/530] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:17.263471 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22da195-ee9f-4d17-9b28-6303d1f2bb6e None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:17.317819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9377686c-aa5f-484b-8be7-b7b99b60c42e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:17.343100 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:17.353756 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf91f23d-f00d-450f-a73d-f8828429d560 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:17.354395 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf91f23d-f00d-450f-a73d-f8828429d560 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:17.354677 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf91f23d-f00d-450f-a73d-f8828429d560 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:17.354957 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf91f23d-f00d-450f-a73d-f8828429d560 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:17.365550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf91f23d-f00d-450f-a73d-f8828429d560 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:17.367008 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf91f23d-f00d-450f-a73d-f8828429d560 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:17.369677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22da195-ee9f-4d17-9b28-6303d1f2bb6e None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:17.382420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-bf91f23d-f00d-450f-a73d-f8828429d560 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:17.382420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:17.382420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:17.382420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:17.382420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:17.382420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:17.382420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:17.382420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:17.382420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:17.382420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:17.382420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:17.382420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:17.382420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:17.382420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:17.382420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:17.382420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:17.383316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:17.383316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:17.383316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:17.383316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:17.383316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:17.383316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:17.383316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:17.383316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:17.383316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:17.383316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:17.383316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:17.383316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:17.383316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:17.383316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:17.383316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:17.383316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:17.383316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:17.384149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:17.384149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:17.384149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:17.384149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 263/531] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:17.384372 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf91f23d-f00d-450f-a73d-f8828429d560 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:17.414407 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2747d87b-721c-4350-9b2d-f9333aac3a93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:17.415110 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2747d87b-721c-4350-9b2d-f9333aac3a93 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:17.415390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2747d87b-721c-4350-9b2d-f9333aac3a93 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:17.415664 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2747d87b-721c-4350-9b2d-f9333aac3a93 None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:17.416352 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2747d87b-721c-4350-9b2d-f9333aac3a93 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:17.416656 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2747d87b-721c-4350-9b2d-f9333aac3a93 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:17.418181 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2747d87b-721c-4350-9b2d-f9333aac3a93 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:17.422913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:17.441110 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 269/532] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:17.441110 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2747d87b-721c-4350-9b2d-f9333aac3a93 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:17.498133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf91f23d-f00d-450f-a73d-f8828429d560 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:17.537541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f670a60a-54de-44bc-ba2a-24f61b71676f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:17.538152 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f670a60a-54de-44bc-ba2a-24f61b71676f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:17.538436 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f670a60a-54de-44bc-ba2a-24f61b71676f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:17.538703 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f670a60a-54de-44bc-ba2a-24f61b71676f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:17.539433 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f670a60a-54de-44bc-ba2a-24f61b71676f None admin] RBAC: Adding query filter params (role.id=755eb1bfc0cf445d8f282358f404a320, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=354b8228661c4369b3bca2e631f3f140) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:17.539741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f670a60a-54de-44bc-ba2a-24f61b71676f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:17.541234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f670a60a-54de-44bc-ba2a-24f61b71676f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:17.553067 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 264/533] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:13:17 2022] GET /identity/v3/role_assignments?user.id=354b8228661c4369b3bca2e631f3f140&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=755eb1bfc0cf445d8f282358f404a320 => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:17.553067 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f670a60a-54de-44bc-ba2a-24f61b71676f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:17.553318 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2747d87b-721c-4350-9b2d-f9333aac3a93 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:17.586691 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b1454e-e929-4ac5-bbb4-2b16058335fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:17.587081 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b1454e-e929-4ac5-bbb4-2b16058335fc None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:17.587357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b1454e-e929-4ac5-bbb4-2b16058335fc None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:17.587628 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b1454e-e929-4ac5-bbb4-2b16058335fc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:17.597853 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b1454e-e929-4ac5-bbb4-2b16058335fc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:17.599398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b1454e-e929-4ac5-bbb4-2b16058335fc None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:17.612463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-93b1454e-e929-4ac5-bbb4-2b16058335fc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:17.612463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:17.612463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:17.612463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:17.612463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:17.612463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:17.612463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:17.612463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:17.612463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:17.612463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:17.612463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:17.612463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:17.612463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:17.612463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:17.612463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:17.613429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:17.613429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:17.613429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:17.613429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:17.613429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:17.613429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:17.613429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:17.613429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:17.613429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:17.613429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:17.613429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:17.613429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:17.613429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:17.613429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:17.613429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:17.613429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:17.613429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:17.614673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:17.614673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:17.614673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:17.614673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:17.614673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:17.614673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:17.614673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:17.614673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:17.614673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 270/534] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:17.615246 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b1454e-e929-4ac5-bbb4-2b16058335fc None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:17.615700 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:17.698067 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:17.704878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f670a60a-54de-44bc-ba2a-24f61b71676f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:17.743981 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b1454e-e929-4ac5-bbb4-2b16058335fc None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:17.753169 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81fdc24-6cf1-4dfa-bffb-d86115470f83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:17.754437 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81fdc24-6cf1-4dfa-bffb-d86115470f83 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:17.754437 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81fdc24-6cf1-4dfa-bffb-d86115470f83 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:17.754437 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81fdc24-6cf1-4dfa-bffb-d86115470f83 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:17.754989 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81fdc24-6cf1-4dfa-bffb-d86115470f83 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:17.755296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81fdc24-6cf1-4dfa-bffb-d86115470f83 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:17.757009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81fdc24-6cf1-4dfa-bffb-d86115470f83 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:17.773874 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 265/535] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:17.774785 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81fdc24-6cf1-4dfa-bffb-d86115470f83 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:17.796949 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d286ea7-493b-4408-b9a3-78b847b871c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:17.797960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d286ea7-493b-4408-b9a3-78b847b871c3 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:17.798480 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d286ea7-493b-4408-b9a3-78b847b871c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:17.799543 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d286ea7-493b-4408-b9a3-78b847b871c3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:17.800785 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d286ea7-493b-4408-b9a3-78b847b871c3 None admin] RBAC: Adding query filter params (role.id=755eb1bfc0cf445d8f282358f404a320, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=4bc4d455ebf041d488b1199ca58358a4) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:17.801419 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d286ea7-493b-4408-b9a3-78b847b871c3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:17.804576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d286ea7-493b-4408-b9a3-78b847b871c3 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:17.812905 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 271/536] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:13:17 2022] GET /identity/v3/role_assignments?user.id=4bc4d455ebf041d488b1199ca58358a4&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=755eb1bfc0cf445d8f282358f404a320 => generated 259 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:17.814004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d286ea7-493b-4408-b9a3-78b847b871c3 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:17.944434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:17.987778 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81fdc24-6cf1-4dfa-bffb-d86115470f83 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:18.002208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d286ea7-493b-4408-b9a3-78b847b871c3 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:18.028140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c66bd2-14a6-4a41-933e-a57c9cdd966d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:18.028759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c66bd2-14a6-4a41-933e-a57c9cdd966d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:18.029104 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c66bd2-14a6-4a41-933e-a57c9cdd966d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:18.029316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c66bd2-14a6-4a41-933e-a57c9cdd966d None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:18.030025 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c66bd2-14a6-4a41-933e-a57c9cdd966d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:18.030332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c66bd2-14a6-4a41-933e-a57c9cdd966d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:18.031915 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c66bd2-14a6-4a41-933e-a57c9cdd966d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:18.040897 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2f291a-5f1a-4e6d-a1a2-bb76427bc8b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:18.041760 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2f291a-5f1a-4e6d-a1a2-bb76427bc8b8 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:18.042211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2f291a-5f1a-4e6d-a1a2-bb76427bc8b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:18.042629 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2f291a-5f1a-4e6d-a1a2-bb76427bc8b8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:18.050181 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2f291a-5f1a-4e6d-a1a2-bb76427bc8b8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:18.051056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 266/537] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:18.052037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2f291a-5f1a-4e6d-a1a2-bb76427bc8b8 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:18.052435 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c66bd2-14a6-4a41-933e-a57c9cdd966d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:18.064495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-ba2f291a-5f1a-4e6d-a1a2-bb76427bc8b8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:18.064495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:18.064495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:18.064495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:18.064495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:18.064495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:18.064495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:18.064495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:18.064495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:18.064495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:18.064495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:18.064495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:18.064495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:18.064495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:18.064495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:18.065635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:18.065635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:18.065635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:18.065635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:18.065635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:18.065635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:18.065635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:18.065635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:18.065635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:18.065635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:18.065635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:18.065635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:18.065635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:18.065635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:18.065635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:18.065635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:18.065635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:18.066612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:18.066612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:18.066612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:18.066612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:18.066612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:18.066612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:18.066612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:18.066612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:18.068559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 272/538] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:18.074187 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2f291a-5f1a-4e6d-a1a2-bb76427bc8b8 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:18.075385 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:18.183194 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2f291a-5f1a-4e6d-a1a2-bb76427bc8b8 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:18.184132 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c66bd2-14a6-4a41-933e-a57c9cdd966d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:18.207905 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:18.221356 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f8505b-d281-452b-959f-888e42108a74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:18.222159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f8505b-d281-452b-959f-888e42108a74 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:18.222609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f8505b-d281-452b-959f-888e42108a74 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:18.222861 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f65f97-e4f0-4135-a533-910b3979abe8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:18.223205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f8505b-d281-452b-959f-888e42108a74 None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:18.223723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f65f97-e4f0-4135-a533-910b3979abe8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:18.224206 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f65f97-e4f0-4135-a533-910b3979abe8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:18.224439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f8505b-d281-452b-959f-888e42108a74 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:18.224668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f65f97-e4f0-4135-a533-910b3979abe8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:18.225110 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f8505b-d281-452b-959f-888e42108a74 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:18.225614 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f65f97-e4f0-4135-a533-910b3979abe8 None admin] RBAC: Adding query filter params (role.id=755eb1bfc0cf445d8f282358f404a320, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=7ef280ce39b14085be329f60b22530d0) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:18.226012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f65f97-e4f0-4135-a533-910b3979abe8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:18.226984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f8505b-d281-452b-959f-888e42108a74 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:18.227616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f65f97-e4f0-4135-a533-910b3979abe8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:18.233917 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 267/539] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:13:18 2022] GET /identity/v3/role_assignments?user.id=7ef280ce39b14085be329f60b22530d0&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=755eb1bfc0cf445d8f282358f404a320 => generated 259 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:18.235439 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d9b31f-6748-4b1d-8424-e2fb27942196 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:18.235817 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d9b31f-6748-4b1d-8424-e2fb27942196 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:18.236203 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d9b31f-6748-4b1d-8424-e2fb27942196 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:18.242737 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 268/540] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:18.244553 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240a1ee4-c0b8-42db-a532-4c6edac3d2f8 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:18.244724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 273/541] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:18.249225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240a1ee4-c0b8-42db-a532-4c6edac3d2f8 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:18.249225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240a1ee4-c0b8-42db-a532-4c6edac3d2f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:18.251699 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f8505b-d281-452b-959f-888e42108a74 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:18.274396 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-240a1ee4-c0b8-42db-a532-4c6edac3d2f8 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:18.275795 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-240a1ee4-c0b8-42db-a532-4c6edac3d2f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:18.378564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f8505b-d281-452b-959f-888e42108a74 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:18.409062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 269/542] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:18.420140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8eece43-fd6c-45d8-a2d4-fe4c2bb257a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:18.420760 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eece43-fd6c-45d8-a2d4-fe4c2bb257a5 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:18.421064 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eece43-fd6c-45d8-a2d4-fe4c2bb257a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:18.421465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eece43-fd6c-45d8-a2d4-fe4c2bb257a5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:18.422208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8eece43-fd6c-45d8-a2d4-fe4c2bb257a5 None admin] RBAC: Adding query filter params (role.id=755eb1bfc0cf445d8f282358f404a320, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=5c2edd77896a4534888d6758ec0ee0bc) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:18.422564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8eece43-fd6c-45d8-a2d4-fe4c2bb257a5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:18.424020 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8eece43-fd6c-45d8-a2d4-fe4c2bb257a5 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:18.431554 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 274/543] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:13:18 2022] GET /identity/v3/role_assignments?user.id=5c2edd77896a4534888d6758ec0ee0bc&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=755eb1bfc0cf445d8f282358f404a320 => generated 259 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:18.694606 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240a1ee4-c0b8-42db-a532-4c6edac3d2f8 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:18.709853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:18.824315 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240a1ee4-c0b8-42db-a532-4c6edac3d2f8 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:18.872456 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd167d6-2475-453a-904f-dc6007053d1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:18.873145 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd167d6-2475-453a-904f-dc6007053d1a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:18.873375 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd167d6-2475-453a-904f-dc6007053d1a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:18.873638 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd167d6-2475-453a-904f-dc6007053d1a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:18.886700 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd167d6-2475-453a-904f-dc6007053d1a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:18.891620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-0bd167d6-2475-453a-904f-dc6007053d1a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:18.891620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:18.891620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:18.891620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:18.891620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:18.891620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:18.891620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:18.891620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:18.891620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:18.891620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:18.891620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:18.891620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:18.891620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:18.891620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:18.891620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:18.891620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:18.892452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:18.892452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:18.892452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:18.892452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:18.892452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:18.892452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:18.892452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:18.892452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:18.892452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:18.892452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:18.892452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:18.892452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:18.892452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:18.892452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:18.892452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:18.892452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:18.893278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:18.893278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:18.893278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:18.893278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:18.893278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:18.893278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:18.893278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:18.893278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:18.893278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:18.893278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:18.893278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:18.893278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:18.893278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 270/544] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:18 2022] GET /identity/v3/roles/service => generated 85 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:18.902144 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8eece43-fd6c-45d8-a2d4-fe4c2bb257a5 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:18.977135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:19.059739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8eece43-fd6c-45d8-a2d4-fe4c2bb257a5 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:19.109584 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a581651-e00b-4dbe-a1c1-bc4cbe7bc1dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:19.110205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a581651-e00b-4dbe-a1c1-bc4cbe7bc1dd None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:19.110481 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a581651-e00b-4dbe-a1c1-bc4cbe7bc1dd None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:19.110748 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a581651-e00b-4dbe-a1c1-bc4cbe7bc1dd None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:19.111419 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a581651-e00b-4dbe-a1c1-bc4cbe7bc1dd None admin] RBAC: Adding query filter params (name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:19.111774 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a581651-e00b-4dbe-a1c1-bc4cbe7bc1dd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:19.113312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a581651-e00b-4dbe-a1c1-bc4cbe7bc1dd None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:19.125434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 275/545] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:13:18 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:19.131753 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd167d6-2475-453a-904f-dc6007053d1a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:19.232031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:19.270297 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd167d6-2475-453a-904f-dc6007053d1a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:19.307162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7272c43-8380-4bdb-bafc-96073d221dac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:19.308232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7272c43-8380-4bdb-bafc-96073d221dac None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:19.308813 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7272c43-8380-4bdb-bafc-96073d221dac None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:19.309309 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7272c43-8380-4bdb-bafc-96073d221dac None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:19.320598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7272c43-8380-4bdb-bafc-96073d221dac None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:19.322335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7272c43-8380-4bdb-bafc-96073d221dac None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:19.338129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-c7272c43-8380-4bdb-bafc-96073d221dac None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:19.338129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:19.338129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:19.338129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:19.338129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:19.338129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:19.338129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:19.338129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:19.338129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:19.338129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:19.338129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:19.338129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:19.338129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:19.338129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:19.338129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:19.338129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:19.339048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:19.339048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:19.339048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:19.339048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:19.339048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:19.339048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:19.339048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:19.339048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:19.339048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:19.339048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:19.339048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:19.339048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:19.339048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:19.339048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:19.339048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:19.339048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:19.339048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:19.339878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:19.339878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:19.339878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:19.341949 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 271/546] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:19.350115 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a581651-e00b-4dbe-a1c1-bc4cbe7bc1dd None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:19.514570 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a581651-e00b-4dbe-a1c1-bc4cbe7bc1dd None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:19.526539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:19.567669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8cc6044-978a-4e60-bd31-9c6d3367c4ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:19.568286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cc6044-978a-4e60-bd31-9c6d3367c4ce None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:19.568563 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cc6044-978a-4e60-bd31-9c6d3367c4ce None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:19.568835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cc6044-978a-4e60-bd31-9c6d3367c4ce None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:19.569506 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8cc6044-978a-4e60-bd31-9c6d3367c4ce None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:19.569802 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8cc6044-978a-4e60-bd31-9c6d3367c4ce None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:19.571164 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8cc6044-978a-4e60-bd31-9c6d3367c4ce None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:19.588666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 276/547] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:19.594427 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7272c43-8380-4bdb-bafc-96073d221dac None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:19.620221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1072114d-2a80-4f66-9e90-72f2d18169d5 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:19.620553 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1072114d-2a80-4f66-9e90-72f2d18169d5 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:19.620775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1072114d-2a80-4f66-9e90-72f2d18169d5 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:19.622380 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 277/548] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:13:19.627607 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008ad758-40b6-4ace-a170-e24921ac998e None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:19.627883 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008ad758-40b6-4ace-a170-e24921ac998e None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:19.628152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008ad758-40b6-4ace-a170-e24921ac998e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:19.666218 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-008ad758-40b6-4ace-a170-e24921ac998e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:19.669638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-008ad758-40b6-4ace-a170-e24921ac998e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:19.782592 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7272c43-8380-4bdb-bafc-96073d221dac None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:19.793996 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 278/549] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:13:19.824439 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:19.842630 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b472e7-4de7-46ba-aedd-3857b58c2ffb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:19.843619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b472e7-4de7-46ba-aedd-3857b58c2ffb None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:19.844075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b472e7-4de7-46ba-aedd-3857b58c2ffb None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:19.844521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b472e7-4de7-46ba-aedd-3857b58c2ffb None admin] PATH_INFO: `/v3/users/swift` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:19.857897 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b472e7-4de7-46ba-aedd-3857b58c2ffb None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:19.859553 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b472e7-4de7-46ba-aedd-3857b58c2ffb None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:19.873081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-b6b472e7-4de7-46ba-aedd-3857b58c2ffb None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:13:19.873081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:19.873081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:19.873081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:19.873081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:19.873081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:19.873081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:19.873081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:19.873081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:19.873081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:19.873081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:19.873081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:19.873081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:19.873081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:19.873081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:19.873081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:19.874564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:19.874564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:19.874564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:19.874564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:19.874564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:19.874564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:19.874564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:19.874564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:19.874564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:19.874564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:19.874564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:19.874564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:19.874564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:19.874564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:19.874564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:19.874564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:19.874564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:19.874564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:19.875615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:19.875615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:19.875615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:19.875615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:19.875615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:19.875615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:19.875615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:19.875615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:19.875615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:13:19.875615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:19.877704 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 272/550] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:13:19 2022] GET /identity/v3/users/swift => generated 83 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:19.886116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008ad758-40b6-4ace-a170-e24921ac998e None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:19.995671 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:19.999405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008ad758-40b6-4ace-a170-e24921ac998e None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:20.042705 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5157e92b-aaf3-4a18-b79d-24aea6e20672 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:20.043304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5157e92b-aaf3-4a18-b79d-24aea6e20672 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:20.043586 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5157e92b-aaf3-4a18-b79d-24aea6e20672 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:20.043862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5157e92b-aaf3-4a18-b79d-24aea6e20672 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:20.044597 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5157e92b-aaf3-4a18-b79d-24aea6e20672 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:20.044902 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5157e92b-aaf3-4a18-b79d-24aea6e20672 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:20.046395 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5157e92b-aaf3-4a18-b79d-24aea6e20672 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:20.068277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 279/551] 10.222.0.85 () {56 vars in 1135 bytes} [Tue Aug 30 04:13:19 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:20.074230 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b472e7-4de7-46ba-aedd-3857b58c2ffb None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:20.204148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5157e92b-aaf3-4a18-b79d-24aea6e20672 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:20.226607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b472e7-4de7-46ba-aedd-3857b58c2ffb None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:20.272044 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8dd61b-4509-411a-b054-dd10139f1d21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:20.272750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8dd61b-4509-411a-b054-dd10139f1d21 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:20.273052 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8dd61b-4509-411a-b054-dd10139f1d21 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:20.273329 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8dd61b-4509-411a-b054-dd10139f1d21 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:20.320011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5157e92b-aaf3-4a18-b79d-24aea6e20672 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:20.332462 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:20.352132 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8dd61b-4509-411a-b054-dd10139f1d21 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:20.353963 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8dd61b-4509-411a-b054-dd10139f1d21 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:20.367642 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeaea47c-0a49-4e0a-ba1d-033d47e2b6cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:20.368416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeaea47c-0a49-4e0a-ba1d-033d47e2b6cb None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:20.368709 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeaea47c-0a49-4e0a-ba1d-033d47e2b6cb None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:20.369151 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeaea47c-0a49-4e0a-ba1d-033d47e2b6cb None admin] PATH_INFO: `/v3/roles/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:20.382762 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeaea47c-0a49-4e0a-ba1d-033d47e2b6cb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:20.389354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-aeaea47c-0a49-4e0a-ba1d-033d47e2b6cb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:20.389354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:20.389354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:20.389354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:20.389354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:20.389354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:20.389354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:20.389354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:20.389354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:20.389354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:20.389354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:20.389354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:20.389354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:20.389354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:20.389354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:20.389354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:20.390547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:20.390547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:20.390547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:20.390547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:20.390547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:20.390547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:20.390547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:20.390547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:20.390547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:20.390547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:20.390547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:20.390547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:20.390547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:20.390547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:20.390547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:20.390547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:20.391572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:20.391572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:20.391572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:20.391572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:20.391572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:20.391572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:20.391572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:20.391572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:20.391572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:20.391572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:20.391572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:20.391572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:20.393165 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 280/552] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:20.394518 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005fbec7-b530-4253-9b43-11dc7918ddca None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:20.394949 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005fbec7-b530-4253-9b43-11dc7918ddca None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:20.395496 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005fbec7-b530-4253-9b43-11dc7918ddca None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:20.402623 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 281/553] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:20 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:20.402623 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005fbec7-b530-4253-9b43-11dc7918ddca None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:20.423823 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:20.483770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 273/554] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 04:13:20.485339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b7fd1e-7582-4176-a705-bcac7033e665 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:20.485872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b7fd1e-7582-4176-a705-bcac7033e665 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:20.486294 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b7fd1e-7582-4176-a705-bcac7033e665 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:20.513028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-17b7fd1e-7582-4176-a705-bcac7033e665 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:20.514587 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-17b7fd1e-7582-4176-a705-bcac7033e665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:20.539667 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005fbec7-b530-4253-9b43-11dc7918ddca None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:20.591526 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819c5a0c-539a-4f9e-a487-47e430d79044 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:20.592052 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819c5a0c-539a-4f9e-a487-47e430d79044 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:20.592331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819c5a0c-539a-4f9e-a487-47e430d79044 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:20.592916 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819c5a0c-539a-4f9e-a487-47e430d79044 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:20.593933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819c5a0c-539a-4f9e-a487-47e430d79044 None admin] RBAC: Adding query filter params (name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:20.594241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819c5a0c-539a-4f9e-a487-47e430d79044 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:20.595675 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819c5a0c-539a-4f9e-a487-47e430d79044 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:20.598417 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 274/555] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:20.600127 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b7fd1e-7582-4176-a705-bcac7033e665 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:20.606544 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 282/556] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:13:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:20.612192 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819c5a0c-539a-4f9e-a487-47e430d79044 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:20.626090 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:20.699917 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:20.724235 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819c5a0c-539a-4f9e-a487-47e430d79044 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:20.731243 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b7fd1e-7582-4176-a705-bcac7033e665 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:20.760531 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f4a6f9-bbb7-47c4-ac0b-11dfcca9303c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:20.761228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f4a6f9-bbb7-47c4-ac0b-11dfcca9303c None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:20.761506 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f4a6f9-bbb7-47c4-ac0b-11dfcca9303c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:20.761820 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f4a6f9-bbb7-47c4-ac0b-11dfcca9303c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:20.772840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f4a6f9-bbb7-47c4-ac0b-11dfcca9303c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:20.774575 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f4a6f9-bbb7-47c4-ac0b-11dfcca9303c None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:20.777928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ed53e6-3216-4de2-b695-846493f8201e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:20.778816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ed53e6-3216-4de2-b695-846493f8201e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:20.779296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ed53e6-3216-4de2-b695-846493f8201e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:20.779768 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ed53e6-3216-4de2-b695-846493f8201e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:20.790767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-51f4a6f9-bbb7-47c4-ac0b-11dfcca9303c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:20.790767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:20.790767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:20.790767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:20.790767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:20.790767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:20.790767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:20.790767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:20.790767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:20.790767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:20.790767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:20.790767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:20.790767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:20.790767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:20.790767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:20.790767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:20.791754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:20.791754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:20.791754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:20.791754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:20.791754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:20.791754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:20.791754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:20.791754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:20.791754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:20.791754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:20.791754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:20.791754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:20.791754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:20.791754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:20.791754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:20.791754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:20.791754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:20.792615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:20.792615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:20.792615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:20.794924 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ed53e6-3216-4de2-b695-846493f8201e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:20.796498 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 283/557] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:20.797339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ed53e6-3216-4de2-b695-846493f8201e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:20.797738 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af84c92-b6ce-4128-b97e-c774985ab6f7 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:20.798014 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af84c92-b6ce-4128-b97e-c774985ab6f7 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:20.798289 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af84c92-b6ce-4128-b97e-c774985ab6f7 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:20.799972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 284/558] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:20.800664 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af84c92-b6ce-4128-b97e-c774985ab6f7 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:20.817830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-a1ed53e6-3216-4de2-b695-846493f8201e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:20.817830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:20.817830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:20.817830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:20.817830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:20.817830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:20.817830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:20.817830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:20.817830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:20.817830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:20.817830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:20.817830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:20.817830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:20.817830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:20.817830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:20.817830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:20.818928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:20.818928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:20.818928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:20.818928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:20.818928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:20.818928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:20.818928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:20.818928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:20.818928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:20.818928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:20.818928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:20.818928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:20.818928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:20.818928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:20.818928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:20.818928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:20.818928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:20.820437 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:20.820437 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:20.820437 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:20.823764 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 275/559] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:20.825123 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06abc42d-a3f6-4437-8c04-7052106099d1 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:20.825469 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06abc42d-a3f6-4437-8c04-7052106099d1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:20.825750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06abc42d-a3f6-4437-8c04-7052106099d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:20.846222 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-06abc42d-a3f6-4437-8c04-7052106099d1 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:20.847838 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-06abc42d-a3f6-4437-8c04-7052106099d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:20.913679 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:20.922158 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af84c92-b6ce-4128-b97e-c774985ab6f7 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:20.922782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 276/560] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:20.923899 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06abc42d-a3f6-4437-8c04-7052106099d1 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:20.956300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:20.963896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403f9d9b-7d25-4681-a4da-1a950a0f8b87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:20.964691 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403f9d9b-7d25-4681-a4da-1a950a0f8b87 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:20.965172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403f9d9b-7d25-4681-a4da-1a950a0f8b87 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:20.965618 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403f9d9b-7d25-4681-a4da-1a950a0f8b87 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:20.966414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403f9d9b-7d25-4681-a4da-1a950a0f8b87 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:20.966866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403f9d9b-7d25-4681-a4da-1a950a0f8b87 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:20.968383 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403f9d9b-7d25-4681-a4da-1a950a0f8b87 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:20.981938 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 285/561] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:20.983202 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403f9d9b-7d25-4681-a4da-1a950a0f8b87 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:21.050730 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06abc42d-a3f6-4437-8c04-7052106099d1 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:21.086339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4bac0f-0b21-479c-9df3-11986054cec7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:21.086953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4bac0f-0b21-479c-9df3-11986054cec7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:21.087230 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4bac0f-0b21-479c-9df3-11986054cec7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:21.087501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4bac0f-0b21-479c-9df3-11986054cec7 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:21.088143 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4bac0f-0b21-479c-9df3-11986054cec7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:21.088447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4bac0f-0b21-479c-9df3-11986054cec7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:21.089843 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4bac0f-0b21-479c-9df3-11986054cec7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:21.090675 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403f9d9b-7d25-4681-a4da-1a950a0f8b87 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:21.108307 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 277/562] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:21.108307 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4bac0f-0b21-479c-9df3-11986054cec7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:21.135238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a65c620-ae51-479a-a81d-2e07af48ed16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:21.135812 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a65c620-ae51-479a-a81d-2e07af48ed16 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:21.136088 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a65c620-ae51-479a-a81d-2e07af48ed16 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:21.136356 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a65c620-ae51-479a-a81d-2e07af48ed16 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:21.148750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a65c620-ae51-479a-a81d-2e07af48ed16 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:21.153953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-5a65c620-ae51-479a-a81d-2e07af48ed16 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:21.153953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:21.153953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:21.153953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:21.153953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:21.153953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:21.153953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:21.153953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:21.153953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:21.153953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:21.153953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:21.153953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:21.153953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:21.153953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:21.153953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:21.153953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:21.155004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:21.155004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:21.155004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:21.155004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:21.155004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:21.155004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:21.155004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:21.155004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:21.155004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:21.155004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:21.155004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:21.155004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:21.155004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:21.155004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:21.155004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:21.155004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:21.155874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:21.155874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:21.155874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:21.155874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:21.155874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:21.155874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:21.155874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:21.155874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:21.155874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:21.155874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:21.155874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:21.155874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:21.155874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 286/563] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:21.157093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b525a4-27ca-491b-a252-e43d62e0371d None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:21.157093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b525a4-27ca-491b-a252-e43d62e0371d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:21.157093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b525a4-27ca-491b-a252-e43d62e0371d None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:21.163258 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 287/564] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:21.163258 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b525a4-27ca-491b-a252-e43d62e0371d None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:21.174121 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:21.216989 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:21.221290 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4bac0f-0b21-479c-9df3-11986054cec7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:21.256729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29bcd42-57ce-4d5c-baf6-fffd3a657ca7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:21.257424 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29bcd42-57ce-4d5c-baf6-fffd3a657ca7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:21.257660 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29bcd42-57ce-4d5c-baf6-fffd3a657ca7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:21.257934 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29bcd42-57ce-4d5c-baf6-fffd3a657ca7 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:21.265691 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29bcd42-57ce-4d5c-baf6-fffd3a657ca7 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:21.267165 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29bcd42-57ce-4d5c-baf6-fffd3a657ca7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:21.267719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b525a4-27ca-491b-a252-e43d62e0371d None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:21.280505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-d29bcd42-57ce-4d5c-baf6-fffd3a657ca7 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 30 04:13:21.280505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:21.280505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:21.280505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:21.280505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:21.280505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:21.280505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:21.280505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:21.280505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:21.280505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:21.280505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:21.280505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:21.280505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:21.280505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:21.280505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:21.280505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:21.281515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:21.281515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:21.281515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:21.281515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:21.281515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:21.281515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:21.281515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:21.281515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:21.281515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:21.281515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:21.281515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:21.281515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:21.281515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:21.281515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:21.281515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:21.281515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:21.281515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:21.281515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:21.282438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:21.282438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:21.282438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:21.282438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:21.282438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:21.282438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:21.282438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:21.282438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:21.282438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 30 04:13:21.282438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:21.282966 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 278/565] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:13:21 2022] GET /identity/v3/users/glance => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:21.283951 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29bcd42-57ce-4d5c-baf6-fffd3a657ca7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:21.304990 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb16457e-f307-45b1-a6ea-8340f31831b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:21.305649 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb16457e-f307-45b1-a6ea-8340f31831b9 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:21.305928 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb16457e-f307-45b1-a6ea-8340f31831b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:21.306201 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb16457e-f307-45b1-a6ea-8340f31831b9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:21.312908 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb16457e-f307-45b1-a6ea-8340f31831b9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:21.314403 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb16457e-f307-45b1-a6ea-8340f31831b9 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:21.325660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-cb16457e-f307-45b1-a6ea-8340f31831b9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:21.325660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:21.325660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:21.325660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:21.325660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:21.325660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:21.325660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:21.325660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:21.325660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:21.325660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:21.325660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:21.325660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:21.325660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:21.325660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:21.325660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:21.326532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:21.326532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:21.326532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:21.326532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:21.326532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:21.326532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:21.326532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:21.326532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:21.326532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:21.326532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:21.326532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:21.326532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:21.326532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:21.326532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:21.326532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:21.326532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:21.326532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:21.327390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:21.327390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:21.327390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:21.327390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:21.327390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:21.327390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:21.327390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:21.327390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:21.327390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 288/566] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:21.328150 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c487cf87-951b-426c-99be-8d5701434f59 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:21.328240 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c487cf87-951b-426c-99be-8d5701434f59 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:21.328487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c487cf87-951b-426c-99be-8d5701434f59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:21.347306 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-c487cf87-951b-426c-99be-8d5701434f59 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:21.348499 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c487cf87-951b-426c-99be-8d5701434f59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:21.390110 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29bcd42-57ce-4d5c-baf6-fffd3a657ca7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:21.405590 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:21.417985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 289/567] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:21.418858 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c487cf87-951b-426c-99be-8d5701434f59 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:21.435227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e2525e-5a4d-43d2-962c-394e1bd49de7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:21.435888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e2525e-5a4d-43d2-962c-394e1bd49de7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:21.436173 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e2525e-5a4d-43d2-962c-394e1bd49de7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:21.436525 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e2525e-5a4d-43d2-962c-394e1bd49de7 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:21.437238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e2525e-5a4d-43d2-962c-394e1bd49de7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:21.437596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e2525e-5a4d-43d2-962c-394e1bd49de7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:21.439015 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e2525e-5a4d-43d2-962c-394e1bd49de7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:21.447093 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 279/568] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:13:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:21.447876 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e2525e-5a4d-43d2-962c-394e1bd49de7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:21.463746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:21.528982 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c487cf87-951b-426c-99be-8d5701434f59 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:21.558280 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e2525e-5a4d-43d2-962c-394e1bd49de7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:21.568399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91014f61-576a-4933-bb66-3b71a263d6f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:21.569031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91014f61-576a-4933-bb66-3b71a263d6f8 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:21.569336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91014f61-576a-4933-bb66-3b71a263d6f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:21.569797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91014f61-576a-4933-bb66-3b71a263d6f8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:21.580040 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91014f61-576a-4933-bb66-3b71a263d6f8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:21.585223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-91014f61-576a-4933-bb66-3b71a263d6f8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:21.585223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:21.585223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:21.585223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:21.585223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:21.585223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:21.585223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:21.585223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:21.585223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:21.585223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:21.585223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:21.585223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:21.585223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:21.585223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:21.585223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:21.585223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:21.586135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:21.586135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:21.586135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:21.586135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:21.586135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:21.586135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:21.586135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:21.586135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:21.586135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:21.586135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:21.586135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:21.586135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:21.586135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:21.586135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:21.586135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:21.586135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:21.586988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:21.586988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:21.586988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:21.586988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:21.586988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:21.586988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:21.586988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:21.586988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:21.586988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:21.586988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:21.586988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:21.586988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:21.586988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 290/569] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:21.587676 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91014f61-576a-4933-bb66-3b71a263d6f8 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:21.600373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13060ebf-05c9-41f5-a05c-f56ed7be38df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:21.600991 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13060ebf-05c9-41f5-a05c-f56ed7be38df None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:21.601303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13060ebf-05c9-41f5-a05c-f56ed7be38df None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:21.601571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13060ebf-05c9-41f5-a05c-f56ed7be38df None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:21.602295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13060ebf-05c9-41f5-a05c-f56ed7be38df None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:21.602597 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13060ebf-05c9-41f5-a05c-f56ed7be38df None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:21.604028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13060ebf-05c9-41f5-a05c-f56ed7be38df None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:21.620974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 280/570] 10.222.0.85 () {56 vars in 1137 bytes} [Tue Aug 30 04:13:21 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:21.621650 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13060ebf-05c9-41f5-a05c-f56ed7be38df None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:21.659530 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:21.701671 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:21.704728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91014f61-576a-4933-bb66-3b71a263d6f8 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:21.725430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13060ebf-05c9-41f5-a05c-f56ed7be38df None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:21.738301 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d06a51d-bb49-4112-baa3-c706bec77f2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:21.738902 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d06a51d-bb49-4112-baa3-c706bec77f2a None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:21.739177 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d06a51d-bb49-4112-baa3-c706bec77f2a None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:21.739948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d06a51d-bb49-4112-baa3-c706bec77f2a None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:21.740632 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d06a51d-bb49-4112-baa3-c706bec77f2a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:21.740935 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d06a51d-bb49-4112-baa3-c706bec77f2a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:21.742398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d06a51d-bb49-4112-baa3-c706bec77f2a None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:21.753680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 291/571] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:21.754646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d06a51d-bb49-4112-baa3-c706bec77f2a None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:21.762898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc2fe28-c93d-4666-b1eb-be2d3df1d637 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:21.763537 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc2fe28-c93d-4666-b1eb-be2d3df1d637 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:21.763818 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc2fe28-c93d-4666-b1eb-be2d3df1d637 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:21.764090 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc2fe28-c93d-4666-b1eb-be2d3df1d637 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:21.775649 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc2fe28-c93d-4666-b1eb-be2d3df1d637 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:21.777142 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc2fe28-c93d-4666-b1eb-be2d3df1d637 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:21.791458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-6fc2fe28-c93d-4666-b1eb-be2d3df1d637 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:21.791458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:21.791458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:21.791458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:21.791458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:21.791458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:21.791458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:21.791458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:21.791458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:21.791458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:21.791458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:21.791458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:21.791458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:21.791458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:21.791458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:21.791458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:21.792516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:21.792516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:21.792516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:21.792516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:21.792516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:21.792516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:21.792516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:21.792516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:21.792516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:21.792516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:21.792516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:21.792516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:21.792516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:21.792516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:21.792516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:21.792516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:21.792516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:21.793596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:21.793596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:21.793596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:21.793596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 281/572] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:21.793596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc2fe28-c93d-4666-b1eb-be2d3df1d637 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:21.857023 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d06a51d-bb49-4112-baa3-c706bec77f2a None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:21.882219 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:21.891456 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36944f00-91af-4999-b7c7-4adc59b8a01c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:21.892069 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36944f00-91af-4999-b7c7-4adc59b8a01c None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:21.892350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36944f00-91af-4999-b7c7-4adc59b8a01c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:21.892624 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36944f00-91af-4999-b7c7-4adc59b8a01c None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:21.893321 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36944f00-91af-4999-b7c7-4adc59b8a01c None admin] RBAC: Adding query filter params (name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:21.893625 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36944f00-91af-4999-b7c7-4adc59b8a01c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:21.893818 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc2fe28-c93d-4666-b1eb-be2d3df1d637 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:21.894993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36944f00-91af-4999-b7c7-4adc59b8a01c None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:21.903169 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 292/573] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:13:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:21.903731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36944f00-91af-4999-b7c7-4adc59b8a01c None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:21.927868 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af321a5d-297f-44f0-84e3-282af484886a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:21.928360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:21.928492 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af321a5d-297f-44f0-84e3-282af484886a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:21.928974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af321a5d-297f-44f0-84e3-282af484886a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:21.928974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af321a5d-297f-44f0-84e3-282af484886a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:21.939797 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af321a5d-297f-44f0-84e3-282af484886a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:21.941287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af321a5d-297f-44f0-84e3-282af484886a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:21.955593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-af321a5d-297f-44f0-84e3-282af484886a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:21.955593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:21.955593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:21.955593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:21.955593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:21.955593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:21.955593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:21.955593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:21.955593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:21.955593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:21.955593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:21.955593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:21.955593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:21.955593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:21.955593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:21.955593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:21.956694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:21.956694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:21.956694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:21.956694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:21.956694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:21.956694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:21.956694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:21.956694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:21.956694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:21.956694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:21.956694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:21.956694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:21.956694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:21.956694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:21.956694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:21.956694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:21.956694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:21.957624 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:21.957624 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:21.957624 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:21.957624 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 282/574] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:21.957624 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af321a5d-297f-44f0-84e3-282af484886a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:22.011692 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36944f00-91af-4999-b7c7-4adc59b8a01c None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:22.045769 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4aa602c-ecb6-4f33-bdad-7d71a60b2472 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:22.046400 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aa602c-ecb6-4f33-bdad-7d71a60b2472 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:22.046681 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aa602c-ecb6-4f33-bdad-7d71a60b2472 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:22.046951 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aa602c-ecb6-4f33-bdad-7d71a60b2472 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:22.057569 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4aa602c-ecb6-4f33-bdad-7d71a60b2472 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:22.062462 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af321a5d-297f-44f0-84e3-282af484886a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:22.062839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-d4aa602c-ecb6-4f33-bdad-7d71a60b2472 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:22.062839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:22.062839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:22.062839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:22.062839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:22.062839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:22.062839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:22.062839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:22.062839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:22.062839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:22.062839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:22.062839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:22.062839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:22.062839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:22.062839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:22.062839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:22.063872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:22.063872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:22.063872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:22.063872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:22.063872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:22.063872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:22.063872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:22.063872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:22.063872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:22.063872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:22.063872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:22.063872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:22.063872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:22.063872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:22.063872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:22.063872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:22.064842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:22.064842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:22.064842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:22.064842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:22.064842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:22.064842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:22.064842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:22.064842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:22.064842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:22.064842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:22.064842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:22.064842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:22.064842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 293/575] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:22.064842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4aa602c-ecb6-4f33-bdad-7d71a60b2472 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:22.099269 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a65b17-853a-447d-9e79-f9af39cd8c66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:22.099875 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a65b17-853a-447d-9e79-f9af39cd8c66 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:22.100154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a65b17-853a-447d-9e79-f9af39cd8c66 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:22.100426 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a65b17-853a-447d-9e79-f9af39cd8c66 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:22.101181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a65b17-853a-447d-9e79-f9af39cd8c66 None admin] RBAC: Adding query filter params (role.id=755eb1bfc0cf445d8f282358f404a320, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=6cb539ef6a69406baa70f660e1a815b4) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:22.101488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a65b17-853a-447d-9e79-f9af39cd8c66 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:22.102930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a65b17-853a-447d-9e79-f9af39cd8c66 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:22.108516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 283/576] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:13:21 2022] GET /identity/v3/role_assignments?user.id=6cb539ef6a69406baa70f660e1a815b4&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=755eb1bfc0cf445d8f282358f404a320 => generated 259 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:22.109354 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a65b17-853a-447d-9e79-f9af39cd8c66 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:22.122894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:22.164199 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:22.186357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4aa602c-ecb6-4f33-bdad-7d71a60b2472 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:22.225116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff04beba-f857-4dcb-ba2e-bc258fcf3d45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:22.225746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff04beba-f857-4dcb-ba2e-bc258fcf3d45 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:22.226013 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff04beba-f857-4dcb-ba2e-bc258fcf3d45 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:22.226278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff04beba-f857-4dcb-ba2e-bc258fcf3d45 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:22.226923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff04beba-f857-4dcb-ba2e-bc258fcf3d45 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:22.227225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff04beba-f857-4dcb-ba2e-bc258fcf3d45 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:22.228625 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff04beba-f857-4dcb-ba2e-bc258fcf3d45 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:22.236441 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a65b17-853a-447d-9e79-f9af39cd8c66 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:22.247075 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 294/577] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:22.247075 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff04beba-f857-4dcb-ba2e-bc258fcf3d45 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:22.279259 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe66fcca-e98c-4ad1-97d2-accb04fac941 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:22.280337 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe66fcca-e98c-4ad1-97d2-accb04fac941 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:22.280751 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe66fcca-e98c-4ad1-97d2-accb04fac941 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:22.281026 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe66fcca-e98c-4ad1-97d2-accb04fac941 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:22.293295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe66fcca-e98c-4ad1-97d2-accb04fac941 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:22.294784 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe66fcca-e98c-4ad1-97d2-accb04fac941 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:22.308953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-fe66fcca-e98c-4ad1-97d2-accb04fac941 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:22.308953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:22.308953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:22.308953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:22.308953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:22.308953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:22.308953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:22.308953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:22.308953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:22.308953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:22.308953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:22.308953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:22.308953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:22.308953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:22.308953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:22.308953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:22.310157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:22.310157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:22.310157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:22.310157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:22.310157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:22.310157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:22.310157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:22.310157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:22.310157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:22.310157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:22.310157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:22.310157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:22.310157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:22.310157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:22.310157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:22.310157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:22.310157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:22.311242 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:22.311242 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:22.311242 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:22.314686 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 284/578] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:22.315409 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe66fcca-e98c-4ad1-97d2-accb04fac941 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:22.359765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff04beba-f857-4dcb-ba2e-bc258fcf3d45 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:22.377227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:22.398791 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb35874c-5bfb-49ea-a806-bc29e4ad0675 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:22.399669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb35874c-5bfb-49ea-a806-bc29e4ad0675 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:22.400115 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb35874c-5bfb-49ea-a806-bc29e4ad0675 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:22.400386 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb35874c-5bfb-49ea-a806-bc29e4ad0675 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:22.401050 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb35874c-5bfb-49ea-a806-bc29e4ad0675 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:22.401369 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb35874c-5bfb-49ea-a806-bc29e4ad0675 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:22.402735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb35874c-5bfb-49ea-a806-bc29e4ad0675 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:22.418615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 295/579] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:22.418615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb35874c-5bfb-49ea-a806-bc29e4ad0675 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:22.418615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:22.432788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe66fcca-e98c-4ad1-97d2-accb04fac941 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:22.469563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7514dcf-a006-4477-b12c-f4cc8cc5c62c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:22.470091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7514dcf-a006-4477-b12c-f4cc8cc5c62c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:22.470373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7514dcf-a006-4477-b12c-f4cc8cc5c62c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:22.470646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7514dcf-a006-4477-b12c-f4cc8cc5c62c None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:22.471429 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7514dcf-a006-4477-b12c-f4cc8cc5c62c None admin] RBAC: Adding query filter params (name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:22.471736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7514dcf-a006-4477-b12c-f4cc8cc5c62c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:22.473248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7514dcf-a006-4477-b12c-f4cc8cc5c62c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:22.486365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 285/580] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:13:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:22.486365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7514dcf-a006-4477-b12c-f4cc8cc5c62c None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:22.528581 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb35874c-5bfb-49ea-a806-bc29e4ad0675 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:22.571181 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c605e91-bc54-4a43-bc14-a6395839b4ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:22.571810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c605e91-bc54-4a43-bc14-a6395839b4ce None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:22.572088 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c605e91-bc54-4a43-bc14-a6395839b4ce None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:22.572358 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c605e91-bc54-4a43-bc14-a6395839b4ce None admin] PATH_INFO: `/v3/users/nova` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:22.582083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c605e91-bc54-4a43-bc14-a6395839b4ce None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:22.583598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c605e91-bc54-4a43-bc14-a6395839b4ce None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:22.594158 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7514dcf-a006-4477-b12c-f4cc8cc5c62c None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:22.595119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-0c605e91-bc54-4a43-bc14-a6395839b4ce None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:13:22.595119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:22.595119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:22.595119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:22.595119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:22.595119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:22.595119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:22.595119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:22.595119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:22.595119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:22.595119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:22.595119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:22.595119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:22.595119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:22.595119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:22.595119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:22.596030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:22.596030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:22.596030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:22.596030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:22.596030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:22.596030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:22.596030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:22.596030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:22.596030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:22.596030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:22.596030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:22.596030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:22.596030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:22.596030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:22.596030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:22.596030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:22.596030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:22.596030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:22.596975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:22.596975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:22.596975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:22.596975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:22.596975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:22.596975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:22.596975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:22.596975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:22.596975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:13:22.596975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:22.596975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 296/581] 10.222.0.85 () {56 vars in 1093 bytes} [Tue Aug 30 04:13:22 2022] GET /identity/v3/users/nova => generated 82 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:22.596975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c605e91-bc54-4a43-bc14-a6395839b4ce None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:22.617013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:22.637341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b318f2-ebae-4ec7-bfb9-a24bca83aeb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:22.638082 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b318f2-ebae-4ec7-bfb9-a24bca83aeb8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:22.638597 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b318f2-ebae-4ec7-bfb9-a24bca83aeb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:22.638939 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b318f2-ebae-4ec7-bfb9-a24bca83aeb8 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:22.639976 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b318f2-ebae-4ec7-bfb9-a24bca83aeb8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:22.640868 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b318f2-ebae-4ec7-bfb9-a24bca83aeb8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:22.642272 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b318f2-ebae-4ec7-bfb9-a24bca83aeb8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:22.655385 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 286/582] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:22.660545 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b318f2-ebae-4ec7-bfb9-a24bca83aeb8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:22.664199 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:22.708878 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c605e91-bc54-4a43-bc14-a6395839b4ce None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:22.748484 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10ebe25-69af-4380-930f-b61d9bae119e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:22.749074 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10ebe25-69af-4380-930f-b61d9bae119e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:22.749516 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10ebe25-69af-4380-930f-b61d9bae119e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:22.749645 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10ebe25-69af-4380-930f-b61d9bae119e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:22.757834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10ebe25-69af-4380-930f-b61d9bae119e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:22.759244 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10ebe25-69af-4380-930f-b61d9bae119e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:22.768767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b318f2-ebae-4ec7-bfb9-a24bca83aeb8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:22.770717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-e10ebe25-69af-4380-930f-b61d9bae119e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:22.770717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:22.770717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:22.770717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:22.770717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:22.770717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:22.770717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:22.770717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:22.770717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:22.770717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:22.770717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:22.770717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:22.770717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:22.770717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:22.770717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:22.771574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:22.771574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:22.771574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:22.771574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:22.771574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:22.771574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:22.771574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:22.771574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:22.771574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:22.771574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:22.771574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:22.771574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:22.771574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:22.771574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:22.771574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:22.771574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:22.771574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:22.772457 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:22.772457 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:22.772457 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:22.772457 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:22.772457 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:22.772457 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:22.772457 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:22.772457 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:22.772457 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 297/583] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:22.777116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10ebe25-69af-4380-930f-b61d9bae119e None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:22.807346 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b122c2-7a54-4b76-aaf8-b0d033d8706b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:22.807986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b122c2-7a54-4b76-aaf8-b0d033d8706b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:22.808267 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b122c2-7a54-4b76-aaf8-b0d033d8706b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:22.808598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b122c2-7a54-4b76-aaf8-b0d033d8706b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:22.819292 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b122c2-7a54-4b76-aaf8-b0d033d8706b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:22.820754 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b122c2-7a54-4b76-aaf8-b0d033d8706b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:22.835567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-06b122c2-7a54-4b76-aaf8-b0d033d8706b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:22.835567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:22.835567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:22.835567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:22.835567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:22.835567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:22.835567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:22.835567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:22.835567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:22.835567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:22.835567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:22.835567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:22.835567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:22.835567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:22.835567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:22.835567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:22.836510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:22.836510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:22.836510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:22.836510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:22.836510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:22.836510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:22.836510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:22.836510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:22.836510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:22.836510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:22.836510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:22.836510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:22.836510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:22.836510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:22.836510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:22.836510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:22.836510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:22.837418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:22.837418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:22.837418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:22.837418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 287/584] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:22.838165 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b122c2-7a54-4b76-aaf8-b0d033d8706b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:22.870540 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:22.891681 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10ebe25-69af-4380-930f-b61d9bae119e None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:22.917570 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:22.930728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e0d977-bccd-4a64-9a86-d8a27bdbeacf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:22.931339 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e0d977-bccd-4a64-9a86-d8a27bdbeacf None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:22.931622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e0d977-bccd-4a64-9a86-d8a27bdbeacf None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:22.931893 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e0d977-bccd-4a64-9a86-d8a27bdbeacf None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:22.932622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e0d977-bccd-4a64-9a86-d8a27bdbeacf None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:22.932929 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e0d977-bccd-4a64-9a86-d8a27bdbeacf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:22.934433 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e0d977-bccd-4a64-9a86-d8a27bdbeacf None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:22.944034 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b122c2-7a54-4b76-aaf8-b0d033d8706b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:22.953094 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 298/585] 10.222.0.85 () {56 vars in 1133 bytes} [Tue Aug 30 04:13:22 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:22.953921 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e0d977-bccd-4a64-9a86-d8a27bdbeacf None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:22.983152 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3648111-c658-4798-aa9e-1e2500c2f616 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:22.983795 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3648111-c658-4798-aa9e-1e2500c2f616 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:22.984079 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3648111-c658-4798-aa9e-1e2500c2f616 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:22.984351 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3648111-c658-4798-aa9e-1e2500c2f616 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:22.992474 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3648111-c658-4798-aa9e-1e2500c2f616 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:22.993969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3648111-c658-4798-aa9e-1e2500c2f616 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:23.005227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-c3648111-c658-4798-aa9e-1e2500c2f616 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 04:13:23.005227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:23.005227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:23.005227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:23.005227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:23.005227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:23.005227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:23.005227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:23.005227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:23.005227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:23.005227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:23.005227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:23.005227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:23.005227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:23.005227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:23.005227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:23.006139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:23.006139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:23.006139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:23.006139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:23.006139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:23.006139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:23.006139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:23.006139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:23.006139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:23.006139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:23.006139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:23.006139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:23.006139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:23.006139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:23.006139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:23.006139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:23.006139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:23.006139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:23.007318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:23.007318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:23.007318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:23.007318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:23.007318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:23.007318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:23.007318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:23.007318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:23.007318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 04:13:23.007318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:23.007318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 288/586] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:13:22 2022] GET /identity/v3/users/cinder => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:23.007884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3648111-c658-4798-aa9e-1e2500c2f616 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:23.062644 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e0d977-bccd-4a64-9a86-d8a27bdbeacf None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:23.099422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9d79cb-4108-4d5d-8341-f06182678ae2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:23.100027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9d79cb-4108-4d5d-8341-f06182678ae2 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:23.100779 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9d79cb-4108-4d5d-8341-f06182678ae2 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:23.100887 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:23.100991 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9d79cb-4108-4d5d-8341-f06182678ae2 None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:23.101288 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9d79cb-4108-4d5d-8341-f06182678ae2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:23.101693 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9d79cb-4108-4d5d-8341-f06182678ae2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:23.103125 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9d79cb-4108-4d5d-8341-f06182678ae2 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:23.116439 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3648111-c658-4798-aa9e-1e2500c2f616 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:23.119454 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 299/587] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:23.120593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9d79cb-4108-4d5d-8341-f06182678ae2 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:23.158257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fee2b6e-8cb8-4f7f-adb5-e6b8c7f945fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:23.159315 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fee2b6e-8cb8-4f7f-adb5-e6b8c7f945fb None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:23.159826 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fee2b6e-8cb8-4f7f-adb5-e6b8c7f945fb None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:23.160294 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fee2b6e-8cb8-4f7f-adb5-e6b8c7f945fb None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:23.161254 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fee2b6e-8cb8-4f7f-adb5-e6b8c7f945fb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:23.161766 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fee2b6e-8cb8-4f7f-adb5-e6b8c7f945fb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:23.163322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fee2b6e-8cb8-4f7f-adb5-e6b8c7f945fb None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:23.164672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:23.182703 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 289/588] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:23.182703 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fee2b6e-8cb8-4f7f-adb5-e6b8c7f945fb None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:23.235668 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9d79cb-4108-4d5d-8341-f06182678ae2 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:23.272274 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b26125e-66dc-4f6b-9815-0226224cbe3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:23.272831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b26125e-66dc-4f6b-9815-0226224cbe3a None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:23.273212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b26125e-66dc-4f6b-9815-0226224cbe3a None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:23.273490 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b26125e-66dc-4f6b-9815-0226224cbe3a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:23.284445 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b26125e-66dc-4f6b-9815-0226224cbe3a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:23.285909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b26125e-66dc-4f6b-9815-0226224cbe3a None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:23.289125 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fee2b6e-8cb8-4f7f-adb5-e6b8c7f945fb None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:23.307059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-6b26125e-66dc-4f6b-9815-0226224cbe3a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:23.307059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:23.307059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:23.307059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:23.307059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:23.307059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:23.307059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:23.307059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:23.307059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:23.307059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:23.307059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:23.307059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:23.307059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:23.307059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:23.307059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:23.307059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:23.309168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:23.309168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:23.309168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:23.309168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:23.309168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:23.309168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:23.309168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:23.309168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:23.309168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:23.309168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:23.309168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:23.309168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:23.309168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:23.309168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:23.309168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:23.309168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:23.309168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:23.311243 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:23.311243 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:23.311243 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:23.318035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 300/589] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:23.319023 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b26125e-66dc-4f6b-9815-0226224cbe3a None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:23.325118 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4270ba2b-93e2-4c6e-95b2-390c3e0bf26c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:23.325740 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4270ba2b-93e2-4c6e-95b2-390c3e0bf26c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:23.326025 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4270ba2b-93e2-4c6e-95b2-390c3e0bf26c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:23.326334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4270ba2b-93e2-4c6e-95b2-390c3e0bf26c None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:23.327073 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4270ba2b-93e2-4c6e-95b2-390c3e0bf26c None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:23.327379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4270ba2b-93e2-4c6e-95b2-390c3e0bf26c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:23.328813 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4270ba2b-93e2-4c6e-95b2-390c3e0bf26c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:23.347065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 290/590] 10.222.0.85 () {56 vars in 1137 bytes} [Tue Aug 30 04:13:23 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:23.347996 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4270ba2b-93e2-4c6e-95b2-390c3e0bf26c None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:23.360046 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:23.420521 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:23.435091 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b26125e-66dc-4f6b-9815-0226224cbe3a None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:23.460114 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4270ba2b-93e2-4c6e-95b2-390c3e0bf26c None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:23.470885 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29a865c-3568-45b4-a25a-d0d8e6d6bf58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:23.471550 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29a865c-3568-45b4-a25a-d0d8e6d6bf58 None admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:23.471833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29a865c-3568-45b4-a25a-d0d8e6d6bf58 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:23.472161 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29a865c-3568-45b4-a25a-d0d8e6d6bf58 None admin] PATH_INFO: `/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/354b8228661c4369b3bca2e631f3f140/roles/755eb1bfc0cf445d8f282358f404a320` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:23.488382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29a865c-3568-45b4-a25a-d0d8e6d6bf58 None admin] RBAC: Authorizing `identity:create_grant(project_id=a860dcc3b2d64d1491c40710ec7423a5, user_id=354b8228661c4369b3bca2e631f3f140, role_id=755eb1bfc0cf445d8f282358f404a320)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:23.490393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29a865c-3568-45b4-a25a-d0d8e6d6bf58 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:23.495396 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca601919-a0a9-43df-9d71-f641eda295cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:23.496093 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca601919-a0a9-43df-9d71-f641eda295cf None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:23.496389 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca601919-a0a9-43df-9d71-f641eda295cf None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:23.496696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca601919-a0a9-43df-9d71-f641eda295cf None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:23.513073 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 301/591] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:13:23 2022] PUT /identity/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/354b8228661c4369b3bca2e631f3f140/roles/755eb1bfc0cf445d8f282358f404a320 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:13:23.513293 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca601919-a0a9-43df-9d71-f641eda295cf None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:23.513418 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29a865c-3568-45b4-a25a-d0d8e6d6bf58 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:23.513509 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca601919-a0a9-43df-9d71-f641eda295cf None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:23.527012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-ca601919-a0a9-43df-9d71-f641eda295cf None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 30 04:13:23.527012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:23.527012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:23.527012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:23.527012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:23.527012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:23.527012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:23.527012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:23.527012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:23.527012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:23.527012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:23.527012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:23.527012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:23.527012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:23.527012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:23.527012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:23.528164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:23.528164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:23.528164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:23.528164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:23.528164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:23.528164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:23.528164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:23.528164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:23.528164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:23.528164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:23.528164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:23.528164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:23.528164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:23.528164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:23.528164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:23.528164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:23.528164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:23.528164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:23.529260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:23.529260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:23.529260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:23.529260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:23.529260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:23.529260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:23.529260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:23.529260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:23.529260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 30 04:13:23.529260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:23.529260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 291/592] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:23 2022] GET /identity/v3/users/neutron => generated 85 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:23.533991 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca601919-a0a9-43df-9d71-f641eda295cf None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:23.616039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:23.657444 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca601919-a0a9-43df-9d71-f641eda295cf None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:23.658389 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29a865c-3568-45b4-a25a-d0d8e6d6bf58 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:23.698978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e25560d-6e92-4868-8bf0-dbcb22120450 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:23.699937 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e25560d-6e92-4868-8bf0-dbcb22120450 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:23.700222 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e25560d-6e92-4868-8bf0-dbcb22120450 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:23.700501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e25560d-6e92-4868-8bf0-dbcb22120450 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:23.705590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942e7b0f-6107-40ae-aa33-8aa56954dd0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:23.706534 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942e7b0f-6107-40ae-aa33-8aa56954dd0f None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:23.706534 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942e7b0f-6107-40ae-aa33-8aa56954dd0f None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:23.706732 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942e7b0f-6107-40ae-aa33-8aa56954dd0f None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:23.707852 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942e7b0f-6107-40ae-aa33-8aa56954dd0f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:23.708210 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942e7b0f-6107-40ae-aa33-8aa56954dd0f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:23.709639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942e7b0f-6107-40ae-aa33-8aa56954dd0f None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:23.712068 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e25560d-6e92-4868-8bf0-dbcb22120450 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:23.713926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e25560d-6e92-4868-8bf0-dbcb22120450 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:23.724469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 302/593] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:23.725377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942e7b0f-6107-40ae-aa33-8aa56954dd0f None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:23.728063 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:23.731772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-1e25560d-6e92-4868-8bf0-dbcb22120450 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:23.731772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:23.731772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:23.731772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:23.731772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:23.731772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:23.731772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:23.731772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:23.731772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:23.731772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:23.731772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:23.731772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:23.731772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:23.731772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:23.731772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:23.731772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:23.732840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:23.732840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:23.732840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:23.732840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:23.732840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:23.732840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:23.732840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:23.732840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:23.732840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:23.732840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:23.732840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:23.732840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:23.732840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:23.732840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:23.732840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:23.732840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:23.732840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:23.733763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:23.733763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:23.733763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:23.739264 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 292/594] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:23.739264 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e25560d-6e92-4868-8bf0-dbcb22120450 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:23.840724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942e7b0f-6107-40ae-aa33-8aa56954dd0f None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:23.858741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e25560d-6e92-4868-8bf0-dbcb22120450 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:23.881100 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:23.893209 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655b2f7c-02c5-4318-8cd8-0fb09f109403 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:23.893818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655b2f7c-02c5-4318-8cd8-0fb09f109403 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:23.894148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655b2f7c-02c5-4318-8cd8-0fb09f109403 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:23.894423 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655b2f7c-02c5-4318-8cd8-0fb09f109403 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:23.895152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655b2f7c-02c5-4318-8cd8-0fb09f109403 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:23.895992 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655b2f7c-02c5-4318-8cd8-0fb09f109403 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:23.897540 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655b2f7c-02c5-4318-8cd8-0fb09f109403 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:23.903575 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d190956-07e6-4f96-8382-1e7740fc3a32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:23.904523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d190956-07e6-4f96-8382-1e7740fc3a32 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:23.905266 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d190956-07e6-4f96-8382-1e7740fc3a32 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:23.905729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d190956-07e6-4f96-8382-1e7740fc3a32 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:23.915254 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d190956-07e6-4f96-8382-1e7740fc3a32 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:23.916953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d190956-07e6-4f96-8382-1e7740fc3a32 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:23.920779 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 303/595] 10.222.0.85 () {56 vars in 1139 bytes} [Tue Aug 30 04:13:23 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:23.921789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655b2f7c-02c5-4318-8cd8-0fb09f109403 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:23.932911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-1d190956-07e6-4f96-8382-1e7740fc3a32 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:23.932911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:23.932911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:23.932911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:23.932911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:23.932911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:23.932911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:23.932911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:23.932911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:23.932911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:23.932911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:23.932911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:23.932911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:23.932911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:23.932911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:23.933871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:23.933871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:23.933871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:23.933871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:23.933871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:23.933871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:23.933871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:23.933871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:23.933871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:23.933871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:23.933871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:23.933871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:23.933871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:23.933871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:23.933871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:23.933871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:23.933871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:23.934748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:23.934748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:23.934748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:23.934748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:23.934748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:23.934748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:23.934748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:23.934748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:23.941841 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 293/596] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:23.941841 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d190956-07e6-4f96-8382-1e7740fc3a32 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:23.993523 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:24.044310 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655b2f7c-02c5-4318-8cd8-0fb09f109403 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:24.078101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d190956-07e6-4f96-8382-1e7740fc3a32 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:24.081390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7ad265-a38a-440f-a82e-2afc1429de37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:24.082005 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7ad265-a38a-440f-a82e-2afc1429de37 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:24.082287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7ad265-a38a-440f-a82e-2afc1429de37 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:24.082558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7ad265-a38a-440f-a82e-2afc1429de37 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:24.083729 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7ad265-a38a-440f-a82e-2afc1429de37 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:24.084028 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7ad265-a38a-440f-a82e-2afc1429de37 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:24.085417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7ad265-a38a-440f-a82e-2afc1429de37 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:24.097808 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 304/597] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:24.098457 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7ad265-a38a-440f-a82e-2afc1429de37 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:24.122582 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9da80dc-6d83-4289-80ef-abc0eb511c88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:24.123216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9da80dc-6d83-4289-80ef-abc0eb511c88 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:24.123498 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9da80dc-6d83-4289-80ef-abc0eb511c88 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:24.123773 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9da80dc-6d83-4289-80ef-abc0eb511c88 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:24.135154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9da80dc-6d83-4289-80ef-abc0eb511c88 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:24.136671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9da80dc-6d83-4289-80ef-abc0eb511c88 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:24.151652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-d9da80dc-6d83-4289-80ef-abc0eb511c88 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:24.151652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:24.151652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:24.151652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:24.151652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:24.151652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:24.151652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:24.151652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:24.151652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:24.151652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:24.151652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:24.151652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:24.151652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:24.151652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:24.151652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:24.151652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:24.152631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:24.152631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:24.152631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:24.152631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:24.152631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:24.152631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:24.152631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:24.152631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:24.152631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:24.152631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:24.152631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:24.152631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:24.152631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:24.152631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:24.152631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:24.152631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:24.152631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:24.153562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:24.153562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:24.153562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:24.153562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 294/598] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:24.154343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d9b138-0ae5-431a-9736-2b3d06f99eb2 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:24.154343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d9b138-0ae5-431a-9736-2b3d06f99eb2 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:24.154513 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d9b138-0ae5-431a-9736-2b3d06f99eb2 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:24.155811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 295/599] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:24 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:24.156348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d9b138-0ae5-431a-9736-2b3d06f99eb2 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:24.190935 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:24.221974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7ad265-a38a-440f-a82e-2afc1429de37 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:24.244513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:24.262756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0418c2db-c16f-4aca-871c-742687d26956 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:24.263367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0418c2db-c16f-4aca-871c-742687d26956 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:24.263648 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0418c2db-c16f-4aca-871c-742687d26956 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:24.263923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0418c2db-c16f-4aca-871c-742687d26956 None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:24.264614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0418c2db-c16f-4aca-871c-742687d26956 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:24.264921 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0418c2db-c16f-4aca-871c-742687d26956 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:24.266464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0418c2db-c16f-4aca-871c-742687d26956 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:24.272505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d9b138-0ae5-431a-9736-2b3d06f99eb2 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:24.279325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 305/600] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:24.280018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0418c2db-c16f-4aca-871c-742687d26956 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:24.313544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6cf1e6-cdf0-468e-b9ad-238bd2400854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:24.314335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6cf1e6-cdf0-468e-b9ad-238bd2400854 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:24.314619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6cf1e6-cdf0-468e-b9ad-238bd2400854 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:24.314892 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6cf1e6-cdf0-468e-b9ad-238bd2400854 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:24.322737 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6cf1e6-cdf0-468e-b9ad-238bd2400854 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:24.324205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6cf1e6-cdf0-468e-b9ad-238bd2400854 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:24.336361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-8a6cf1e6-cdf0-468e-b9ad-238bd2400854 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:24.336361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:24.336361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:24.336361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:24.336361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:24.336361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:24.336361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:24.336361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:24.336361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:24.336361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:24.336361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:24.336361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:24.336361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:24.336361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:24.336361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:24.337690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:24.337690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:24.337690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:24.337690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:24.337690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:24.337690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:24.337690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:24.337690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:24.337690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:24.337690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:24.337690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:24.337690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:24.337690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:24.337690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:24.337690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:24.337690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:24.337690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:24.338593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:24.338593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:24.338593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:24.338593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:24.338593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:24.338593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:24.338593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:24.338593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:24.338593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 296/601] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:24.339933 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b64d55-eb43-4576-8097-8a458b6dcb24 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:24.340443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b64d55-eb43-4576-8097-8a458b6dcb24 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:24.341006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b64d55-eb43-4576-8097-8a458b6dcb24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:24.361598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b7b64d55-eb43-4576-8097-8a458b6dcb24 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:24.362757 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b7b64d55-eb43-4576-8097-8a458b6dcb24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:24.398356 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0418c2db-c16f-4aca-871c-742687d26956 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:24.437415 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a035fd07-bc28-45ef-ae40-97f8c978969e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:24.437806 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 297/602] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:24.438627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b64d55-eb43-4576-8097-8a458b6dcb24 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:24.438739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a035fd07-bc28-45ef-ae40-97f8c978969e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:24.439223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a035fd07-bc28-45ef-ae40-97f8c978969e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:24.439704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a035fd07-bc28-45ef-ae40-97f8c978969e None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:24.440622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a035fd07-bc28-45ef-ae40-97f8c978969e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:24.441179 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a035fd07-bc28-45ef-ae40-97f8c978969e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:24.442664 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a035fd07-bc28-45ef-ae40-97f8c978969e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:24.455868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 306/603] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:24.457177 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a035fd07-bc28-45ef-ae40-97f8c978969e None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:24.480626 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:24.518760 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:24.568847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a035fd07-bc28-45ef-ae40-97f8c978969e None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:24.575434 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b64d55-eb43-4576-8097-8a458b6dcb24 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:24.613587 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fe5e8f-4b68-4f6e-a418-d6de11881b27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:24.615195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fe5e8f-4b68-4f6e-a418-d6de11881b27 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:24.615682 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fe5e8f-4b68-4f6e-a418-d6de11881b27 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:24.615789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8afb64-0b1c-45a8-89ea-bf2beba0a9fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:24.616310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fe5e8f-4b68-4f6e-a418-d6de11881b27 None admin] PATH_INFO: `/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/4bc4d455ebf041d488b1199ca58358a4/roles/755eb1bfc0cf445d8f282358f404a320` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:24.616417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8afb64-0b1c-45a8-89ea-bf2beba0a9fe None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:24.616638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8afb64-0b1c-45a8-89ea-bf2beba0a9fe None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:24.616972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8afb64-0b1c-45a8-89ea-bf2beba0a9fe None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:24.617832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8afb64-0b1c-45a8-89ea-bf2beba0a9fe None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:24.618140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8afb64-0b1c-45a8-89ea-bf2beba0a9fe None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:24.619746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8afb64-0b1c-45a8-89ea-bf2beba0a9fe None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:24.634638 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70fe5e8f-4b68-4f6e-a418-d6de11881b27 None admin] RBAC: Authorizing `identity:create_grant(project_id=a860dcc3b2d64d1491c40710ec7423a5, user_id=4bc4d455ebf041d488b1199ca58358a4, role_id=755eb1bfc0cf445d8f282358f404a320)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:24.636410 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70fe5e8f-4b68-4f6e-a418-d6de11881b27 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:24.639542 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 307/604] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:24.642285 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8afb64-0b1c-45a8-89ea-bf2beba0a9fe None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:24.661303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 298/605] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:13:24 2022] PUT /identity/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/4bc4d455ebf041d488b1199ca58358a4/roles/755eb1bfc0cf445d8f282358f404a320 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:13:24.662791 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fe5e8f-4b68-4f6e-a418-d6de11881b27 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:24.753288 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:24.788980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fe5e8f-4b68-4f6e-a418-d6de11881b27 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:24.797815 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:24.803811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8afb64-0b1c-45a8-89ea-bf2beba0a9fe None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:24.828895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305c990c-dce0-4b6e-bacd-660e0d56e146 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:24.829797 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305c990c-dce0-4b6e-bacd-660e0d56e146 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:24.830175 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305c990c-dce0-4b6e-bacd-660e0d56e146 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:24.830559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305c990c-dce0-4b6e-bacd-660e0d56e146 None admin] PATH_INFO: `/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/7ef280ce39b14085be329f60b22530d0/roles/755eb1bfc0cf445d8f282358f404a320` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:24.847705 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305c990c-dce0-4b6e-bacd-660e0d56e146 None admin] RBAC: Authorizing `identity:create_grant(project_id=a860dcc3b2d64d1491c40710ec7423a5, user_id=7ef280ce39b14085be329f60b22530d0, role_id=755eb1bfc0cf445d8f282358f404a320)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:24.849378 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305c990c-dce0-4b6e-bacd-660e0d56e146 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:24.854087 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed50a3e-2c7a-4005-9af9-b6c1746e96af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:24.855131 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed50a3e-2c7a-4005-9af9-b6c1746e96af None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:24.855665 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed50a3e-2c7a-4005-9af9-b6c1746e96af None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:24.856265 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed50a3e-2c7a-4005-9af9-b6c1746e96af None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:24.866183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed50a3e-2c7a-4005-9af9-b6c1746e96af None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:24.868347 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed50a3e-2c7a-4005-9af9-b6c1746e96af None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:24.869349 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 299/606] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:13:24 2022] PUT /identity/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/7ef280ce39b14085be329f60b22530d0/roles/755eb1bfc0cf445d8f282358f404a320 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:13:24.870432 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305c990c-dce0-4b6e-bacd-660e0d56e146 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:24.883220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-aed50a3e-2c7a-4005-9af9-b6c1746e96af None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:24.883220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:24.883220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:24.883220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:24.883220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:24.883220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:24.883220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:24.883220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:24.883220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:24.883220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:24.883220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:24.883220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:24.883220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:24.883220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:24.883220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:24.884176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:24.884176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:24.884176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:24.884176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:24.884176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:24.884176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:24.884176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:24.884176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:24.884176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:24.884176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:24.884176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:24.884176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:24.884176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:24.884176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:24.884176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:24.884176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:24.884176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:24.886223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:24.886223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:24.886223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:24.886223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:24.886223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:24.886223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:24.886223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:24.886223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:24.888710 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 308/607] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:24.895402 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed50a3e-2c7a-4005-9af9-b6c1746e96af None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:25.005669 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305c990c-dce0-4b6e-bacd-660e0d56e146 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:25.027670 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed50a3e-2c7a-4005-9af9-b6c1746e96af None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:25.046781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:25.052572 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73dd6d1-8935-4620-87df-5aee3a5103dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:25.053695 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73dd6d1-8935-4620-87df-5aee3a5103dd None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:25.054187 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73dd6d1-8935-4620-87df-5aee3a5103dd None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:25.054646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73dd6d1-8935-4620-87df-5aee3a5103dd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:25.071449 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893d3f0a-a040-4a7f-8080-56c94336b154 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:25.072065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893d3f0a-a040-4a7f-8080-56c94336b154 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:25.072344 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893d3f0a-a040-4a7f-8080-56c94336b154 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:25.072619 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893d3f0a-a040-4a7f-8080-56c94336b154 None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:25.073632 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893d3f0a-a040-4a7f-8080-56c94336b154 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:25.073946 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893d3f0a-a040-4a7f-8080-56c94336b154 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:25.074068 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73dd6d1-8935-4620-87df-5aee3a5103dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:25.075376 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893d3f0a-a040-4a7f-8080-56c94336b154 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:25.079365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-f73dd6d1-8935-4620-87df-5aee3a5103dd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:25.079365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:25.079365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:25.079365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:25.079365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:25.079365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:25.079365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:25.079365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:25.079365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:25.079365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:25.079365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:25.079365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:25.079365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:25.079365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:25.079365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:25.079365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:25.080300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:25.080300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:25.080300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:25.080300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:25.080300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:25.080300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:25.080300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:25.080300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:25.080300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:25.080300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:25.080300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:25.080300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:25.080300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:25.080300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:25.080300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:25.080300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:25.081228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:25.081228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:25.081228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:25.081228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:25.081228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:25.081228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:25.081228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:25.081228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:25.081228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:25.081228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:25.081228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:25.081228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:25.082016 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 300/608] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:24 2022] GET /identity/v3/roles/service => generated 85 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:25.089065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 309/609] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:25.089529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73dd6d1-8935-4620-87df-5aee3a5103dd None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:25.096546 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893d3f0a-a040-4a7f-8080-56c94336b154 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:25.125367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:25.216576 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73dd6d1-8935-4620-87df-5aee3a5103dd None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:25.240817 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893d3f0a-a040-4a7f-8080-56c94336b154 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:25.287281 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e163239a-acda-4623-8b51-4a35ac64177d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:25.288296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e163239a-acda-4623-8b51-4a35ac64177d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:25.288782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e163239a-acda-4623-8b51-4a35ac64177d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:25.289311 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e163239a-acda-4623-8b51-4a35ac64177d None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:25.290208 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e163239a-acda-4623-8b51-4a35ac64177d None admin] RBAC: Adding query filter params (name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:25.291365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e163239a-acda-4623-8b51-4a35ac64177d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:25.292952 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e163239a-acda-4623-8b51-4a35ac64177d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:25.310167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 301/610] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:13:25 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:25.311149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21446b2d-2823-4b25-a4f7-7f043a04d752 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:25.313083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21446b2d-2823-4b25-a4f7-7f043a04d752 None admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:25.316766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21446b2d-2823-4b25-a4f7-7f043a04d752 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:25.317073 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21446b2d-2823-4b25-a4f7-7f043a04d752 None admin] PATH_INFO: `/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/5c2edd77896a4534888d6758ec0ee0bc/roles/755eb1bfc0cf445d8f282358f404a320` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:25.318733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e163239a-acda-4623-8b51-4a35ac64177d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:25.349383 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21446b2d-2823-4b25-a4f7-7f043a04d752 None admin] RBAC: Authorizing `identity:create_grant(project_id=a860dcc3b2d64d1491c40710ec7423a5, user_id=5c2edd77896a4534888d6758ec0ee0bc, role_id=755eb1bfc0cf445d8f282358f404a320)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:25.350839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21446b2d-2823-4b25-a4f7-7f043a04d752 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:25.370770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 310/611] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:13:25 2022] PUT /identity/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/5c2edd77896a4534888d6758ec0ee0bc/roles/755eb1bfc0cf445d8f282358f404a320 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:13:25.407628 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:25.466930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e163239a-acda-4623-8b51-4a35ac64177d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:25.513383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a32ab1-eb09-4efa-ada2-1680edd41bd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:25.514049 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a32ab1-eb09-4efa-ada2-1680edd41bd5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:25.514333 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a32ab1-eb09-4efa-ada2-1680edd41bd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:25.514607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a32ab1-eb09-4efa-ada2-1680edd41bd5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:25.527400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a32ab1-eb09-4efa-ada2-1680edd41bd5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:25.528918 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a32ab1-eb09-4efa-ada2-1680edd41bd5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:25.544975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-b1a32ab1-eb09-4efa-ada2-1680edd41bd5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:25.544975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:25.544975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:25.544975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:25.544975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:25.544975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:25.544975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:25.544975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:25.544975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:25.544975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:25.544975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:25.544975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:25.544975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:25.544975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:25.544975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:25.544975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:25.546010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:25.546010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:25.546010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:25.546010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:25.546010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:25.546010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:25.546010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:25.546010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:25.546010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:25.546010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:25.546010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:25.546010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:25.546010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:25.546010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:25.546010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:25.546010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:25.546010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:25.547991 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:25.547991 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:25.547991 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:25.547991 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 302/612] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:25.552975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21446b2d-2823-4b25-a4f7-7f043a04d752 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:25.603388 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9776ead4-4227-469b-9113-feb575541899 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:25.604009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9776ead4-4227-469b-9113-feb575541899 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:25.604488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9776ead4-4227-469b-9113-feb575541899 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:25.606212 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 303/613] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:13:25.611439 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1cd5e6-b31c-4f0d-b0df-57d88504b741 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:25.611972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1cd5e6-b31c-4f0d-b0df-57d88504b741 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:25.612448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1cd5e6-b31c-4f0d-b0df-57d88504b741 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:25.633378 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3f1cd5e6-b31c-4f0d-b0df-57d88504b741 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:25.635266 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3f1cd5e6-b31c-4f0d-b0df-57d88504b741 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:25.639587 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:25.677571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21446b2d-2823-4b25-a4f7-7f043a04d752 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:25.716414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88030e7e-a3f2-4a9d-904f-ff2ca9c22a4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:25.717252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88030e7e-a3f2-4a9d-904f-ff2ca9c22a4f None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:25.717604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88030e7e-a3f2-4a9d-904f-ff2ca9c22a4f None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:25.717953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88030e7e-a3f2-4a9d-904f-ff2ca9c22a4f None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:25.718778 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88030e7e-a3f2-4a9d-904f-ff2ca9c22a4f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:25.719129 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88030e7e-a3f2-4a9d-904f-ff2ca9c22a4f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:25.721283 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88030e7e-a3f2-4a9d-904f-ff2ca9c22a4f None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:25.729571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 304/614] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:13:25.739973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 311/615] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:25.745576 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1cd5e6-b31c-4f0d-b0df-57d88504b741 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:25.790020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:25.870133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1cd5e6-b31c-4f0d-b0df-57d88504b741 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:25.927273 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9feced-6d44-40f4-8a90-d0b6dd6ff374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:25.928062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9feced-6d44-40f4-8a90-d0b6dd6ff374 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:25.928553 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9feced-6d44-40f4-8a90-d0b6dd6ff374 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:25.928833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9feced-6d44-40f4-8a90-d0b6dd6ff374 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:25.938790 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9feced-6d44-40f4-8a90-d0b6dd6ff374 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:25.940231 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9feced-6d44-40f4-8a90-d0b6dd6ff374 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:25.956607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-bb9feced-6d44-40f4-8a90-d0b6dd6ff374 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:13:25.956607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:25.956607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:25.956607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:25.956607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:25.956607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:25.956607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:25.956607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:25.956607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:25.956607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:25.956607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:25.956607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:25.956607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:25.956607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:25.956607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:25.956607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:25.957833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:25.957833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:25.957833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:25.957833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:25.957833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:25.957833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:25.957833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:25.957833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:25.957833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:25.957833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:25.957833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:25.957833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:25.957833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:25.957833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:25.957833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:25.957833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:25.957833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:25.957833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:25.959373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:25.959373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:25.959373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:25.959373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:25.959373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:25.959373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:25.959373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:25.959373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:25.959373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:13:25.959373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:25.959373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 305/616] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:13:25 2022] GET /identity/v3/users/swift => generated 83 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:25.963523 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88030e7e-a3f2-4a9d-904f-ff2ca9c22a4f None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:26.093484 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9feced-6d44-40f4-8a90-d0b6dd6ff374 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:26.133396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88030e7e-a3f2-4a9d-904f-ff2ca9c22a4f None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:26.195205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:26.198474 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:26.208990 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c2b6ce-f6ff-4c5f-b218-4be88a5cd073 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:26.209893 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c2b6ce-f6ff-4c5f-b218-4be88a5cd073 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:26.210374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c2b6ce-f6ff-4c5f-b218-4be88a5cd073 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:26.210847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c2b6ce-f6ff-4c5f-b218-4be88a5cd073 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:26.211803 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c2b6ce-f6ff-4c5f-b218-4be88a5cd073 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:26.212998 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c2b6ce-f6ff-4c5f-b218-4be88a5cd073 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:26.214649 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c2b6ce-f6ff-4c5f-b218-4be88a5cd073 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:26.230879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9feced-6d44-40f4-8a90-d0b6dd6ff374 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:26.247955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 312/617] 10.222.0.85 () {56 vars in 1135 bytes} [Tue Aug 30 04:13:25 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:26.258052 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c2b6ce-f6ff-4c5f-b218-4be88a5cd073 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:26.278920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2280562a-e5e1-4047-8119-e905ebd771a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:26.279558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2280562a-e5e1-4047-8119-e905ebd771a2 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:26.279841 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2280562a-e5e1-4047-8119-e905ebd771a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:26.280113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2280562a-e5e1-4047-8119-e905ebd771a2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:26.302675 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2280562a-e5e1-4047-8119-e905ebd771a2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:26.308381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-2280562a-e5e1-4047-8119-e905ebd771a2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:26.308381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:26.308381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:26.308381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:26.308381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:26.308381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:26.308381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:26.308381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:26.308381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:26.308381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:26.308381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:26.308381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:26.308381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:26.308381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:26.308381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:26.308381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:26.309405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:26.309405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:26.309405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:26.309405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:26.309405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:26.309405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:26.309405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:26.309405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:26.309405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:26.309405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:26.309405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:26.309405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:26.309405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:26.309405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:26.309405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:26.309405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:26.310532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:26.310532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:26.310532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:26.310532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:26.310532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:26.310532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:26.310532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:26.310532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:26.310532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:26.310532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:26.310532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:26.310532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:26.310532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 306/618] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:26.318509 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2280562a-e5e1-4047-8119-e905ebd771a2 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:26.466075 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c2b6ce-f6ff-4c5f-b218-4be88a5cd073 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:26.503495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79069ce-134e-429d-850f-163e12c472b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:26.504279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79069ce-134e-429d-850f-163e12c472b5 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:26.504558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79069ce-134e-429d-850f-163e12c472b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:26.504926 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79069ce-134e-429d-850f-163e12c472b5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:26.515943 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79069ce-134e-429d-850f-163e12c472b5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:26.517615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79069ce-134e-429d-850f-163e12c472b5 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:26.519623 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2280562a-e5e1-4047-8119-e905ebd771a2 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:26.533896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-d79069ce-134e-429d-850f-163e12c472b5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:26.533896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:26.533896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:26.533896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:26.533896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:26.533896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:26.533896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:26.533896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:26.533896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:26.533896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:26.533896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:26.533896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:26.533896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:26.533896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:26.533896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:26.533896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:26.535133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:26.535133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:26.535133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:26.535133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:26.535133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:26.535133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:26.535133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:26.535133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:26.535133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:26.535133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:26.535133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:26.535133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:26.535133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:26.535133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:26.535133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:26.535133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:26.535133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:26.536897 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:26.536897 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:26.536897 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:26.536897 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 313/619] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:26.549057 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79069ce-134e-429d-850f-163e12c472b5 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:26.595738 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e22ed5-ec5d-4a59-aece-5a5c28079be4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:26.596356 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e22ed5-ec5d-4a59-aece-5a5c28079be4 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:26.596734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e22ed5-ec5d-4a59-aece-5a5c28079be4 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:26.597217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e22ed5-ec5d-4a59-aece-5a5c28079be4 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:26.597698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e22ed5-ec5d-4a59-aece-5a5c28079be4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:26.597999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e22ed5-ec5d-4a59-aece-5a5c28079be4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:26.599899 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e22ed5-ec5d-4a59-aece-5a5c28079be4 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:26.624889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 307/620] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:13:26 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:26.625820 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e22ed5-ec5d-4a59-aece-5a5c28079be4 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:26.628988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:26.642707 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:26.735245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e22ed5-ec5d-4a59-aece-5a5c28079be4 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:26.773378 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3357c13-12a4-4f85-a083-c6464fb94807 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:26.774013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3357c13-12a4-4f85-a083-c6464fb94807 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:26.774287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3357c13-12a4-4f85-a083-c6464fb94807 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:26.774555 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3357c13-12a4-4f85-a083-c6464fb94807 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:26.787074 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3357c13-12a4-4f85-a083-c6464fb94807 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:26.788536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3357c13-12a4-4f85-a083-c6464fb94807 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:26.806576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79069ce-134e-429d-850f-163e12c472b5 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:26.808088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-a3357c13-12a4-4f85-a083-c6464fb94807 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:26.808088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:26.808088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:26.808088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:26.808088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:26.808088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:26.808088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:26.808088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:26.808088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:26.808088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:26.808088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:26.808088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:26.808088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:26.808088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:26.808088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:26.808088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:26.808992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:26.808992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:26.808992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:26.808992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:26.808992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:26.808992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:26.808992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:26.808992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:26.808992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:26.808992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:26.808992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:26.808992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:26.808992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:26.808992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:26.808992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:26.808992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:26.808992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:26.810865 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:26.810865 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:26.810865 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:26.810865 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 308/621] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:26.817207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3357c13-12a4-4f85-a083-c6464fb94807 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:26.866327 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616ece5c-f12d-4ed6-b0e1-2a68f04046ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:26.866945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616ece5c-f12d-4ed6-b0e1-2a68f04046ab None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:26.867224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616ece5c-f12d-4ed6-b0e1-2a68f04046ab None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:26.867492 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616ece5c-f12d-4ed6-b0e1-2a68f04046ab None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:26.868150 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616ece5c-f12d-4ed6-b0e1-2a68f04046ab None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:26.868449 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616ece5c-f12d-4ed6-b0e1-2a68f04046ab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:26.869847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616ece5c-f12d-4ed6-b0e1-2a68f04046ab None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:26.890859 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:26.894861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 314/622] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 352 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:26.902758 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616ece5c-f12d-4ed6-b0e1-2a68f04046ab None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:26.995289 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3357c13-12a4-4f85-a083-c6464fb94807 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:27.035125 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:27.037354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616ece5c-f12d-4ed6-b0e1-2a68f04046ab None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:27.075359 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358ecc98-77e3-494a-a5b2-cb6f8db2c5db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:27.076438 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358ecc98-77e3-494a-a5b2-cb6f8db2c5db None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:27.076948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358ecc98-77e3-494a-a5b2-cb6f8db2c5db None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:27.077724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358ecc98-77e3-494a-a5b2-cb6f8db2c5db None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:27.085781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69119bf-8749-4553-856c-2092b89fa9d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:27.086306 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358ecc98-77e3-494a-a5b2-cb6f8db2c5db None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:27.086442 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69119bf-8749-4553-856c-2092b89fa9d4 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:27.086685 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69119bf-8749-4553-856c-2092b89fa9d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:27.086962 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69119bf-8749-4553-856c-2092b89fa9d4 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:27.087623 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69119bf-8749-4553-856c-2092b89fa9d4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:27.087926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69119bf-8749-4553-856c-2092b89fa9d4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:27.088960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358ecc98-77e3-494a-a5b2-cb6f8db2c5db None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:27.089321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69119bf-8749-4553-856c-2092b89fa9d4 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:27.102646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-358ecc98-77e3-494a-a5b2-cb6f8db2c5db None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:27.102646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:27.102646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:27.102646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:27.102646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:27.102646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:27.102646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:27.102646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:27.102646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:27.102646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:27.102646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:27.102646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:27.102646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:27.102646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:27.102646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:27.103596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:27.103596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:27.103596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:27.103596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:27.103596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:27.103596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:27.103596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:27.103596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:27.103596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:27.103596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:27.103596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:27.103596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:27.103596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:27.103596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:27.103596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:27.103596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:27.103596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:27.104494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:27.104494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:27.104494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:27.104494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:27.104494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:27.104494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:27.104494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:27.104494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:27.107376 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 315/623] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:27.113582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358ecc98-77e3-494a-a5b2-cb6f8db2c5db None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:27.119419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 309/624] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:27.121464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69119bf-8749-4553-856c-2092b89fa9d4 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:27.228822 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358ecc98-77e3-494a-a5b2-cb6f8db2c5db None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:27.248441 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69119bf-8749-4553-856c-2092b89fa9d4 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:27.267276 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5cace6-b717-4b24-accf-a07d5debbbfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:27.268613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5cace6-b717-4b24-accf-a07d5debbbfe None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:27.269162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:27.270337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5cace6-b717-4b24-accf-a07d5debbbfe None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:27.271161 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5cace6-b717-4b24-accf-a07d5debbbfe None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:27.272732 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5cace6-b717-4b24-accf-a07d5debbbfe None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:27.273679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5cace6-b717-4b24-accf-a07d5debbbfe None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:27.275464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5cace6-b717-4b24-accf-a07d5debbbfe None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:27.283932 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60243931-d6cb-4cef-b766-41655ff1c8c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:27.284570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60243931-d6cb-4cef-b766-41655ff1c8c5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:27.284849 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60243931-d6cb-4cef-b766-41655ff1c8c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:27.285162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60243931-d6cb-4cef-b766-41655ff1c8c5 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:27.297099 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60243931-d6cb-4cef-b766-41655ff1c8c5 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:27.297099 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60243931-d6cb-4cef-b766-41655ff1c8c5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:27.297333 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 316/625] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:27.298481 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a948796-6519-4418-bf53-fe1a8026ec29 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:27.298765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a948796-6519-4418-bf53-fe1a8026ec29 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:27.299032 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a948796-6519-4418-bf53-fe1a8026ec29 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:27.301305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 317/626] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:27.302884 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a948796-6519-4418-bf53-fe1a8026ec29 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:27.318117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-60243931-d6cb-4cef-b766-41655ff1c8c5 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 30 04:13:27.318117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:27.318117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:27.318117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:27.318117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:27.318117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:27.318117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:27.318117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:27.318117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:27.318117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:27.318117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:27.318117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:27.318117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:27.318117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:27.318117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:27.318117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:27.319232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:27.319232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:27.319232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:27.319232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:27.319232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:27.319232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:27.319232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:27.319232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:27.319232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:27.319232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:27.319232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:27.319232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:27.319232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:27.319232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:27.319232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:27.319232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:27.319232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:27.319232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:27.320211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:27.320211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:27.320211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:27.320211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:27.320211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:27.320211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:27.320211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:27.320211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:27.320211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 30 04:13:27.320211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:27.323998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 310/627] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:13:27 2022] GET /identity/v3/users/glance => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:27.325526 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958fb151-0ec9-4bbe-8d1a-56688b68f57f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:27.325650 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958fb151-0ec9-4bbe-8d1a-56688b68f57f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:27.325920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958fb151-0ec9-4bbe-8d1a-56688b68f57f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:27.345251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-958fb151-0ec9-4bbe-8d1a-56688b68f57f None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:27.346588 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-958fb151-0ec9-4bbe-8d1a-56688b68f57f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:27.371878 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:27.417334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 311/628] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:27.419557 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958fb151-0ec9-4bbe-8d1a-56688b68f57f None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:27.434426 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a948796-6519-4418-bf53-fe1a8026ec29 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:27.486378 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c6b491-79f4-43d5-af73-d5e2710bdaae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:27.487449 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c6b491-79f4-43d5-af73-d5e2710bdaae None admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:27.488026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c6b491-79f4-43d5-af73-d5e2710bdaae None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:27.488712 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c6b491-79f4-43d5-af73-d5e2710bdaae None admin] PATH_INFO: `/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/6cb539ef6a69406baa70f660e1a815b4/roles/755eb1bfc0cf445d8f282358f404a320` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:27.510034 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c6b491-79f4-43d5-af73-d5e2710bdaae None admin] RBAC: Authorizing `identity:create_grant(project_id=a860dcc3b2d64d1491c40710ec7423a5, user_id=6cb539ef6a69406baa70f660e1a815b4, role_id=755eb1bfc0cf445d8f282358f404a320)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:27.512150 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c6b491-79f4-43d5-af73-d5e2710bdaae None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:27.528948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958fb151-0ec9-4bbe-8d1a-56688b68f57f None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:27.537477 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 318/629] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:13:27 2022] PUT /identity/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/6cb539ef6a69406baa70f660e1a815b4/roles/755eb1bfc0cf445d8f282358f404a320 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:13:27.540978 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfb16a0-d5bb-40ec-90b9-6eb5713b5c76 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:27.541485 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfb16a0-d5bb-40ec-90b9-6eb5713b5c76 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:27.542548 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:27.543934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfb16a0-d5bb-40ec-90b9-6eb5713b5c76 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:27.548477 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 319/630] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:27 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:27.554836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e3f1e0-0dbb-4e8d-ac93-8f205d5f7592 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:27.555024 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e3f1e0-0dbb-4e8d-ac93-8f205d5f7592 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:27.555228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e3f1e0-0dbb-4e8d-ac93-8f205d5f7592 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:27.592840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b293d2d7-3213-45cb-b375-bb176fe5f375 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:27.593597 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b293d2d7-3213-45cb-b375-bb176fe5f375 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:27.593877 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b293d2d7-3213-45cb-b375-bb176fe5f375 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:27.594147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b293d2d7-3213-45cb-b375-bb176fe5f375 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:27.594880 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b293d2d7-3213-45cb-b375-bb176fe5f375 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:27.596122 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-f6e3f1e0-0dbb-4e8d-ac93-8f205d5f7592 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:27.596861 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b293d2d7-3213-45cb-b375-bb176fe5f375 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:27.598150 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f6e3f1e0-0dbb-4e8d-ac93-8f205d5f7592 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:27.600568 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b293d2d7-3213-45cb-b375-bb176fe5f375 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:27.639595 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 312/631] 10.222.0.85 () {56 vars in 1137 bytes} [Tue Aug 30 04:13:27 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:27.643033 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b293d2d7-3213-45cb-b375-bb176fe5f375 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:27.711265 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:27.716736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 320/632] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:27.718125 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e3f1e0-0dbb-4e8d-ac93-8f205d5f7592 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:27.782930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b293d2d7-3213-45cb-b375-bb176fe5f375 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:27.821727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3b6539-8ee4-48b9-ad91-16401258f254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:27.822617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3b6539-8ee4-48b9-ad91-16401258f254 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:27.823048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3b6539-8ee4-48b9-ad91-16401258f254 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:27.823473 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3b6539-8ee4-48b9-ad91-16401258f254 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:27.827861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e3f1e0-0dbb-4e8d-ac93-8f205d5f7592 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:27.862540 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:27.871084 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fb5293-9a5b-4a65-af88-f068cfcc7de3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:27.871731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fb5293-9a5b-4a65-af88-f068cfcc7de3 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:27.872010 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fb5293-9a5b-4a65-af88-f068cfcc7de3 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:27.872281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fb5293-9a5b-4a65-af88-f068cfcc7de3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:27.886902 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13fb5293-9a5b-4a65-af88-f068cfcc7de3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:27.893835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-13fb5293-9a5b-4a65-af88-f068cfcc7de3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:27.893835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:27.893835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:27.893835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:27.893835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:27.893835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:27.893835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:27.893835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:27.893835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:27.893835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:27.893835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:27.893835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:27.893835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:27.893835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:27.893835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:27.893835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:27.894960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:27.894960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:27.894960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:27.894960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:27.894960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:27.894960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:27.894960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:27.894960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:27.894960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:27.894960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:27.894960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:27.894960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:27.894960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:27.894960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:27.894960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:27.894960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:27.896037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:27.896037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:27.896037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:27.896037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:27.896037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:27.896037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:27.896037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:27.896037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:27.896037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:27.896037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:27.896037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:27.896037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:27.899973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 321/633] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:27.901670 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d91f24-730f-40e3-8fe4-a09383e0550e None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:27.901965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d91f24-730f-40e3-8fe4-a09383e0550e None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:27.902241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d91f24-730f-40e3-8fe4-a09383e0550e None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:27.905280 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 322/634] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:27.906114 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d91f24-730f-40e3-8fe4-a09383e0550e None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:27.910703 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3b6539-8ee4-48b9-ad91-16401258f254 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:27.912424 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3b6539-8ee4-48b9-ad91-16401258f254 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:27.983701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:28.026983 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 313/635] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:13:28.028470 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e731773-843d-46af-807d-56360d14c844 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:28.028755 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e731773-843d-46af-807d-56360d14c844 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:28.029028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e731773-843d-46af-807d-56360d14c844 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:28.043691 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d91f24-730f-40e3-8fe4-a09383e0550e None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:28.047795 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-8e731773-843d-46af-807d-56360d14c844 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:28.049002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8e731773-843d-46af-807d-56360d14c844 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:28.082585 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c54c264-ce47-4132-842a-40bb6cdfc619 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:28.083222 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c54c264-ce47-4132-842a-40bb6cdfc619 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:28.083506 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c54c264-ce47-4132-842a-40bb6cdfc619 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:28.083780 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c54c264-ce47-4132-842a-40bb6cdfc619 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:28.084566 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c54c264-ce47-4132-842a-40bb6cdfc619 None admin] RBAC: Adding query filter params (name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:28.084869 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c54c264-ce47-4132-842a-40bb6cdfc619 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:28.086512 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c54c264-ce47-4132-842a-40bb6cdfc619 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:28.099961 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 323/636] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:13:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:28.100910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c54c264-ce47-4132-842a-40bb6cdfc619 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:28.122888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:28.129186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 314/637] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:28.130659 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e731773-843d-46af-807d-56360d14c844 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:28.226053 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c54c264-ce47-4132-842a-40bb6cdfc619 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:28.249911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e731773-843d-46af-807d-56360d14c844 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:28.250368 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:28.267865 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afc6dc0-56cf-4fdb-ad71-f463b879567e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:28.268518 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afc6dc0-56cf-4fdb-ad71-f463b879567e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:28.268924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afc6dc0-56cf-4fdb-ad71-f463b879567e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:28.269286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afc6dc0-56cf-4fdb-ad71-f463b879567e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:28.280275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afc6dc0-56cf-4fdb-ad71-f463b879567e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:28.285707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-0afc6dc0-56cf-4fdb-ad71-f463b879567e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:28.285707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:28.285707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:28.285707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:28.285707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:28.285707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:28.285707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:28.285707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:28.285707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:28.285707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:28.285707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:28.285707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:28.285707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:28.285707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:28.285707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:28.285707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:28.287357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:28.287357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:28.287357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:28.287357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:28.287357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:28.287357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:28.287357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:28.287357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:28.287357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:28.287357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:28.287357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:28.287357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:28.287357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:28.287357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:28.287357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:28.287357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:28.288377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:28.288377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:28.288377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:28.288377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:28.288377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:28.288377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:28.288377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:28.288377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:28.288377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:28.288377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:28.288377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:28.288377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:28.293095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 324/638] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:28.293095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afc6dc0-56cf-4fdb-ad71-f463b879567e None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:28.302756 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239b281e-425f-47ea-a681-f0b6b2fd19ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:28.303690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239b281e-425f-47ea-a681-f0b6b2fd19ea None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:28.304152 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239b281e-425f-47ea-a681-f0b6b2fd19ea None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:28.304601 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239b281e-425f-47ea-a681-f0b6b2fd19ea None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:28.315449 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239b281e-425f-47ea-a681-f0b6b2fd19ea None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:28.317193 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239b281e-425f-47ea-a681-f0b6b2fd19ea None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:28.333501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-239b281e-425f-47ea-a681-f0b6b2fd19ea None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:28.333501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:28.333501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:28.333501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:28.333501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:28.333501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:28.333501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:28.333501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:28.333501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:28.333501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:28.333501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:28.333501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:28.333501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:28.333501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:28.333501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:28.333501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:28.334602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:28.334602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:28.334602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:28.334602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:28.334602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:28.334602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:28.334602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:28.334602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:28.334602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:28.334602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:28.334602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:28.334602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:28.334602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:28.334602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:28.334602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:28.334602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:28.334602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:28.335486 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:28.335486 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:28.335486 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:28.337305 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 315/639] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:28.338361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239b281e-425f-47ea-a681-f0b6b2fd19ea None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:28.392420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:28.412770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afc6dc0-56cf-4fdb-ad71-f463b879567e None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:28.446917 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239b281e-425f-47ea-a681-f0b6b2fd19ea None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:28.448325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6a3883-88a0-497b-b1bd-f0356073c2d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:28.449067 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6a3883-88a0-497b-b1bd-f0356073c2d0 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:28.449373 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6a3883-88a0-497b-b1bd-f0356073c2d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:28.449648 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6a3883-88a0-497b-b1bd-f0356073c2d0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:28.461271 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6a3883-88a0-497b-b1bd-f0356073c2d0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:28.462710 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6a3883-88a0-497b-b1bd-f0356073c2d0 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:28.477948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-bc6a3883-88a0-497b-b1bd-f0356073c2d0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:28.477948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:28.477948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:28.477948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:28.477948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:28.477948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:28.477948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:28.477948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:28.477948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:28.477948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:28.477948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:28.477948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:28.477948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:28.477948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:28.477948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:28.477948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:28.478932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:28.478932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:28.478932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:28.478932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:28.478932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:28.478932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:28.478932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:28.478932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:28.478932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:28.478932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:28.478932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:28.478932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:28.478932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:28.478932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:28.478932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:28.478932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:28.478932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:28.479826 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:28.479826 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:28.479826 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:28.479826 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 325/640] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:28.479826 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6a3883-88a0-497b-b1bd-f0356073c2d0 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:28.490263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c6f0e9-b49d-43dd-89f3-eb58ec47bf75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:28.491082 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c6f0e9-b49d-43dd-89f3-eb58ec47bf75 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:28.491533 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c6f0e9-b49d-43dd-89f3-eb58ec47bf75 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:28.491954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c6f0e9-b49d-43dd-89f3-eb58ec47bf75 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:28.492767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c6f0e9-b49d-43dd-89f3-eb58ec47bf75 None admin] RBAC: Adding query filter params (name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:28.493354 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c6f0e9-b49d-43dd-89f3-eb58ec47bf75 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:28.494897 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c6f0e9-b49d-43dd-89f3-eb58ec47bf75 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:28.503670 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 316/641] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:13:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:28.504657 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c6f0e9-b49d-43dd-89f3-eb58ec47bf75 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:28.530347 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:28.605622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6a3883-88a0-497b-b1bd-f0356073c2d0 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:28.611590 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c6f0e9-b49d-43dd-89f3-eb58ec47bf75 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:28.626335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:28.648499 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e57a2b7-3f97-4e48-a4f6-41a5ffbaeb28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:28.649153 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e57a2b7-3f97-4e48-a4f6-41a5ffbaeb28 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:28.649434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e57a2b7-3f97-4e48-a4f6-41a5ffbaeb28 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:28.649706 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e57a2b7-3f97-4e48-a4f6-41a5ffbaeb28 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:28.650019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe99ba03-c86e-4784-985d-e7627dee98bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:28.650362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e57a2b7-3f97-4e48-a4f6-41a5ffbaeb28 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:28.650664 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e57a2b7-3f97-4e48-a4f6-41a5ffbaeb28 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:28.650757 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe99ba03-c86e-4784-985d-e7627dee98bc None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:28.650942 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe99ba03-c86e-4784-985d-e7627dee98bc None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:28.651214 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe99ba03-c86e-4784-985d-e7627dee98bc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:28.652028 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e57a2b7-3f97-4e48-a4f6-41a5ffbaeb28 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:28.663080 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe99ba03-c86e-4784-985d-e7627dee98bc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:28.669536 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 326/642] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:28.669536 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e57a2b7-3f97-4e48-a4f6-41a5ffbaeb28 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:28.669723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-fe99ba03-c86e-4784-985d-e7627dee98bc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:28.669723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:28.669723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:28.669723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:28.669723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:28.669723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:28.669723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:28.669723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:28.669723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:28.669723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:28.669723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:28.669723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:28.669723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:28.669723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:28.669723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:28.669723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:28.670665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:28.670665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:28.670665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:28.670665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:28.670665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:28.670665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:28.670665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:28.670665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:28.670665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:28.670665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:28.670665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:28.670665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:28.670665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:28.670665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:28.670665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:28.670665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:28.671506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:28.671506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:28.671506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:28.671506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:28.671506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:28.671506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:28.671506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:28.671506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:28.671506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:28.671506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:28.671506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:28.671506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:28.671506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 317/643] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:28.671506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe99ba03-c86e-4784-985d-e7627dee98bc None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:28.776216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe99ba03-c86e-4784-985d-e7627dee98bc None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:28.780569 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:28.789211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e57a2b7-3f97-4e48-a4f6-41a5ffbaeb28 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:28.814887 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e4ae0d-2923-4b37-a5d6-94f5d47ecfcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:28.815694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e4ae0d-2923-4b37-a5d6-94f5d47ecfcf None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:28.815839 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e4ae0d-2923-4b37-a5d6-94f5d47ecfcf None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:28.816083 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e4ae0d-2923-4b37-a5d6-94f5d47ecfcf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:28.827028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e4ae0d-2923-4b37-a5d6-94f5d47ecfcf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:28.828475 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e4ae0d-2923-4b37-a5d6-94f5d47ecfcf None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:28.828682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eb88a6-e622-41a7-a1a5-c32de0692270 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:28.829316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eb88a6-e622-41a7-a1a5-c32de0692270 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:28.829595 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eb88a6-e622-41a7-a1a5-c32de0692270 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:28.829905 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eb88a6-e622-41a7-a1a5-c32de0692270 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:28.830554 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9eb88a6-e622-41a7-a1a5-c32de0692270 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:28.830864 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9eb88a6-e622-41a7-a1a5-c32de0692270 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:28.832208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9eb88a6-e622-41a7-a1a5-c32de0692270 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:28.843063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-f9e4ae0d-2923-4b37-a5d6-94f5d47ecfcf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:28.843063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:28.843063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:28.843063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:28.843063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:28.843063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:28.843063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:28.843063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:28.843063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:28.843063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:28.843063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:28.843063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:28.843063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:28.843063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:28.843063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:28.843063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:28.844030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:28.844030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:28.844030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:28.844030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:28.844030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:28.844030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:28.844030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:28.844030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:28.844030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:28.844030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:28.844030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:28.844030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:28.844030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:28.844030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:28.844030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:28.844030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:28.844030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:28.845114 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:28.845114 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:28.845114 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:28.848341 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 327/644] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:28.849407 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 318/645] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:28.849944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eb88a6-e622-41a7-a1a5-c32de0692270 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:28.850063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e4ae0d-2923-4b37-a5d6-94f5d47ecfcf None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:28.876188 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:28.956524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e4ae0d-2923-4b37-a5d6-94f5d47ecfcf None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:28.959316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eb88a6-e622-41a7-a1a5-c32de0692270 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:28.990852 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef346bf-ed16-4114-b0cf-066be1a74900 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:28.991460 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef346bf-ed16-4114-b0cf-066be1a74900 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:28.991740 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef346bf-ed16-4114-b0cf-066be1a74900 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:28.992010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef346bf-ed16-4114-b0cf-066be1a74900 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:28.992666 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef346bf-ed16-4114-b0cf-066be1a74900 None admin] RBAC: Adding query filter params (name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:28.992964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef346bf-ed16-4114-b0cf-066be1a74900 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:28.994365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef346bf-ed16-4114-b0cf-066be1a74900 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:28.998453 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62520df0-5d41-44ad-b926-c8e6666b7023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:28.999488 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62520df0-5d41-44ad-b926-c8e6666b7023 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:28.999780 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62520df0-5d41-44ad-b926-c8e6666b7023 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:29.000052 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62520df0-5d41-44ad-b926-c8e6666b7023 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:29.002200 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 319/646] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:13:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:29.008246 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef346bf-ed16-4114-b0cf-066be1a74900 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:29.008603 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62520df0-5d41-44ad-b926-c8e6666b7023 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:29.010112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62520df0-5d41-44ad-b926-c8e6666b7023 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:29.025002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-62520df0-5d41-44ad-b926-c8e6666b7023 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:29.025002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:29.025002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:29.025002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:29.025002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:29.025002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:29.025002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:29.025002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:29.025002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:29.025002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:29.025002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:29.025002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:29.025002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:29.025002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:29.025002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:29.025933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:29.025933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:29.025933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:29.025933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:29.025933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:29.025933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:29.025933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:29.025933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:29.025933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:29.025933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:29.025933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:29.025933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:29.025933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:29.025933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:29.025933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:29.025933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:29.025933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:29.027483 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:29.027483 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:29.027483 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:29.027483 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:29.027483 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:29.027483 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:29.027483 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:29.027483 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:29.027483 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 328/647] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:29.027483 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62520df0-5d41-44ad-b926-c8e6666b7023 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:29.047562 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:29.108641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:29.126591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef346bf-ed16-4114-b0cf-066be1a74900 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:29.144230 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62520df0-5d41-44ad-b926-c8e6666b7023 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:29.168315 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fed709-0f9e-44d9-a69a-f714673ff7fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:29.169193 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fed709-0f9e-44d9-a69a-f714673ff7fd None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:29.169616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fed709-0f9e-44d9-a69a-f714673ff7fd None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:29.170043 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fed709-0f9e-44d9-a69a-f714673ff7fd None admin] PATH_INFO: `/v3/users/nova` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:29.178906 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35fed709-0f9e-44d9-a69a-f714673ff7fd None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:29.179780 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5cc5ac-0727-40cb-8ec4-a4fac55f29c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:29.180386 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5cc5ac-0727-40cb-8ec4-a4fac55f29c6 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:29.180663 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5cc5ac-0727-40cb-8ec4-a4fac55f29c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:29.180932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5cc5ac-0727-40cb-8ec4-a4fac55f29c6 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:29.181195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35fed709-0f9e-44d9-a69a-f714673ff7fd None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:29.181599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5cc5ac-0727-40cb-8ec4-a4fac55f29c6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:29.181902 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5cc5ac-0727-40cb-8ec4-a4fac55f29c6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:29.183294 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5cc5ac-0727-40cb-8ec4-a4fac55f29c6 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:29.193771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-35fed709-0f9e-44d9-a69a-f714673ff7fd None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:13:29.193771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:29.193771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:29.193771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:29.193771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:29.193771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:29.193771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:29.193771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:29.193771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:29.193771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:29.193771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:29.193771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:29.193771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:29.193771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:29.193771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:29.193771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:29.194799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:29.194799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:29.194799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:29.194799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:29.194799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:29.194799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:29.194799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:29.194799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:29.194799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:29.194799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:29.194799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:29.194799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:29.194799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:29.194799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:29.194799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:29.194799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:29.194799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:29.194799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:29.195934 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:29.195934 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:29.195934 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:29.195934 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:29.195934 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:29.195934 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:29.195934 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:29.195934 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:29.195934 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:13:29.195934 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:29.198026 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 320/648] 10.222.0.85 () {56 vars in 1093 bytes} [Tue Aug 30 04:13:29 2022] GET /identity/v3/users/nova => generated 82 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:29.198960 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fed709-0f9e-44d9-a69a-f714673ff7fd None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:29.201119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 329/649] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:29.202163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5cc5ac-0727-40cb-8ec4-a4fac55f29c6 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:29.281913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:29.305008 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fed709-0f9e-44d9-a69a-f714673ff7fd None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:29.309130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5cc5ac-0727-40cb-8ec4-a4fac55f29c6 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:29.340446 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287c7a20-b1e8-4da8-aa22-42b2de86db83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:29.341094 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287c7a20-b1e8-4da8-aa22-42b2de86db83 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:29.341374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287c7a20-b1e8-4da8-aa22-42b2de86db83 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:29.341646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287c7a20-b1e8-4da8-aa22-42b2de86db83 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:29.344208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08481573-7c8b-448e-babf-809da5f8aa93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:29.344872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08481573-7c8b-448e-babf-809da5f8aa93 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:29.345191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08481573-7c8b-448e-babf-809da5f8aa93 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:29.345465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08481573-7c8b-448e-babf-809da5f8aa93 None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:29.346163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08481573-7c8b-448e-babf-809da5f8aa93 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:29.346469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08481573-7c8b-448e-babf-809da5f8aa93 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:29.347885 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08481573-7c8b-448e-babf-809da5f8aa93 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:29.353197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287c7a20-b1e8-4da8-aa22-42b2de86db83 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:29.354626 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287c7a20-b1e8-4da8-aa22-42b2de86db83 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:29.361464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:29.369470 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 330/650] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:29.369470 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08481573-7c8b-448e-babf-809da5f8aa93 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:29.369980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-287c7a20-b1e8-4da8-aa22-42b2de86db83 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:29.369980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:29.369980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:29.369980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:29.369980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:29.369980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:29.369980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:29.369980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:29.369980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:29.369980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:29.369980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:29.369980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:29.369980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:29.369980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:29.369980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:29.369980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:29.370878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:29.370878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:29.370878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:29.370878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:29.370878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:29.370878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:29.370878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:29.370878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:29.370878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:29.370878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:29.370878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:29.370878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:29.370878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:29.370878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:29.370878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:29.370878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:29.370878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:29.371755 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:29.371755 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:29.371755 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:29.371755 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 321/651] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:29.372519 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287c7a20-b1e8-4da8-aa22-42b2de86db83 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:29.483085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287c7a20-b1e8-4da8-aa22-42b2de86db83 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:29.483658 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08481573-7c8b-448e-babf-809da5f8aa93 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:29.519699 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c90e5f-2ffa-47bd-b6bc-4766f4b892e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:29.520334 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c90e5f-2ffa-47bd-b6bc-4766f4b892e2 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:29.520704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c90e5f-2ffa-47bd-b6bc-4766f4b892e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:29.520857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ccd862-5419-450b-9855-b455a7da8764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:29.521245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c90e5f-2ffa-47bd-b6bc-4766f4b892e2 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:29.521490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ccd862-5419-450b-9855-b455a7da8764 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:29.521729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ccd862-5419-450b-9855-b455a7da8764 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:29.521994 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ccd862-5419-450b-9855-b455a7da8764 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:29.522720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ccd862-5419-450b-9855-b455a7da8764 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:29.523018 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ccd862-5419-450b-9855-b455a7da8764 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:29.524092 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:29.524439 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ccd862-5419-450b-9855-b455a7da8764 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:29.532729 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c90e5f-2ffa-47bd-b6bc-4766f4b892e2 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:29.534700 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c90e5f-2ffa-47bd-b6bc-4766f4b892e2 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:29.543553 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 322/652] 10.222.0.85 () {56 vars in 1133 bytes} [Tue Aug 30 04:13:29 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:29.544632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ccd862-5419-450b-9855-b455a7da8764 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:29.546505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-23c90e5f-2ffa-47bd-b6bc-4766f4b892e2 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 04:13:29.546505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:29.546505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:29.546505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:29.546505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:29.546505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:29.546505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:29.546505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:29.546505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:29.546505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:29.546505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:29.546505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:29.546505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:29.546505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:29.546505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:29.546505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:29.547428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:29.547428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:29.547428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:29.547428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:29.547428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:29.547428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:29.547428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:29.547428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:29.547428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:29.547428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:29.547428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:29.547428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:29.547428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:29.547428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:29.547428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:29.547428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:29.547428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:29.547428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:29.548922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:29.548922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:29.548922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:29.548922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:29.548922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:29.548922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:29.548922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:29.548922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:29.548922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 04:13:29.548922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:29.552944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 331/653] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:13:29 2022] GET /identity/v3/users/cinder => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:29.553814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c90e5f-2ffa-47bd-b6bc-4766f4b892e2 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:29.609639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:29.658914 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ccd862-5419-450b-9855-b455a7da8764 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:29.660191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c90e5f-2ffa-47bd-b6bc-4766f4b892e2 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:29.692922 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3aa5e09-26c6-4937-8c4f-eb00a1a35625 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:29.693378 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9939b123-ff8d-48a1-b117-b2037b839b51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:29.693558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3aa5e09-26c6-4937-8c4f-eb00a1a35625 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:29.693832 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3aa5e09-26c6-4937-8c4f-eb00a1a35625 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:29.693972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9939b123-ff8d-48a1-b117-b2037b839b51 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:29.694098 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3aa5e09-26c6-4937-8c4f-eb00a1a35625 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:29.694248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9939b123-ff8d-48a1-b117-b2037b839b51 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:29.694518 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9939b123-ff8d-48a1-b117-b2037b839b51 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:29.694812 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3aa5e09-26c6-4937-8c4f-eb00a1a35625 None admin] RBAC: Adding query filter params (role.id=755eb1bfc0cf445d8f282358f404a320, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=354b8228661c4369b3bca2e631f3f140) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:29.695113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3aa5e09-26c6-4937-8c4f-eb00a1a35625 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:29.695201 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9939b123-ff8d-48a1-b117-b2037b839b51 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:29.695426 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9939b123-ff8d-48a1-b117-b2037b839b51 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:29.696573 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3aa5e09-26c6-4937-8c4f-eb00a1a35625 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:29.696766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9939b123-ff8d-48a1-b117-b2037b839b51 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:29.702164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 323/654] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:13:29 2022] GET /identity/v3/role_assignments?user.id=354b8228661c4369b3bca2e631f3f140&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=755eb1bfc0cf445d8f282358f404a320 => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:29.702930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3aa5e09-26c6-4937-8c4f-eb00a1a35625 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:29.709193 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 332/655] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:29.710180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9939b123-ff8d-48a1-b117-b2037b839b51 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:29.776378 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:29.828213 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3aa5e09-26c6-4937-8c4f-eb00a1a35625 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:29.828607 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9939b123-ff8d-48a1-b117-b2037b839b51 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:29.852622 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:29.869900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda812ac-710c-4f41-9f93-90762e6fb7ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:29.870523 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda812ac-710c-4f41-9f93-90762e6fb7ee None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:29.870802 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda812ac-710c-4f41-9f93-90762e6fb7ee None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:29.871071 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda812ac-710c-4f41-9f93-90762e6fb7ee None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:29.871800 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda812ac-710c-4f41-9f93-90762e6fb7ee None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:29.872106 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda812ac-710c-4f41-9f93-90762e6fb7ee None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:29.873599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda812ac-710c-4f41-9f93-90762e6fb7ee None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:29.873715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403924c2-fc08-427a-9ef1-30c94e51ed8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:29.874310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403924c2-fc08-427a-9ef1-30c94e51ed8f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:29.874586 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403924c2-fc08-427a-9ef1-30c94e51ed8f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:29.874851 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403924c2-fc08-427a-9ef1-30c94e51ed8f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:29.892982 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 333/656] 10.222.0.85 () {56 vars in 1137 bytes} [Tue Aug 30 04:13:29 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:29.894720 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a542a3-0134-44a8-932b-81cff2cf1630 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:29.895204 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a542a3-0134-44a8-932b-81cff2cf1630 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:29.896231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a542a3-0134-44a8-932b-81cff2cf1630 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:29.898027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 334/657] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:29.899088 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a542a3-0134-44a8-932b-81cff2cf1630 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:29.956659 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403924c2-fc08-427a-9ef1-30c94e51ed8f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:29.958697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403924c2-fc08-427a-9ef1-30c94e51ed8f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:30.021756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a542a3-0134-44a8-932b-81cff2cf1630 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:30.037136 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:30.059326 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703e4374-fdfd-457f-9570-dbe9bcbf10c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:30.060007 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703e4374-fdfd-457f-9570-dbe9bcbf10c3 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:30.060238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703e4374-fdfd-457f-9570-dbe9bcbf10c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:30.060509 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703e4374-fdfd-457f-9570-dbe9bcbf10c3 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:30.062416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 324/658] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:13:30.063252 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403924c2-fc08-427a-9ef1-30c94e51ed8f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:30.072761 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703e4374-fdfd-457f-9570-dbe9bcbf10c3 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:30.074259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703e4374-fdfd-457f-9570-dbe9bcbf10c3 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:30.085805 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:30.086753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-703e4374-fdfd-457f-9570-dbe9bcbf10c3 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 30 04:13:30.086753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:30.086753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:30.086753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:30.086753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:30.086753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:30.086753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:30.086753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:30.086753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:30.086753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:30.086753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:30.086753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:30.086753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:30.086753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:30.086753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:30.086753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:30.087701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:30.087701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:30.087701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:30.087701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:30.087701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:30.087701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:30.087701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:30.087701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:30.087701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:30.087701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:30.087701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:30.087701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:30.087701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:30.087701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:30.087701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:30.087701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:30.087701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:30.087701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:30.088641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:30.088641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:30.088641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:30.088641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:30.088641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:30.088641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:30.088641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:30.088641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:30.088641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 30 04:13:30.088641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:30.088641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 335/659] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:29 2022] GET /identity/v3/users/neutron => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:30.089271 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd265528-a8ce-4ecd-a18b-514d54ece04a None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:30.089340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd265528-a8ce-4ecd-a18b-514d54ece04a None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:30.090067 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd265528-a8ce-4ecd-a18b-514d54ece04a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:30.108083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-bd265528-a8ce-4ecd-a18b-514d54ece04a None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:30.109316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-bd265528-a8ce-4ecd-a18b-514d54ece04a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:30.183803 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 336/660] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:30.184899 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd265528-a8ce-4ecd-a18b-514d54ece04a None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:30.185208 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403924c2-fc08-427a-9ef1-30c94e51ed8f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:30.225933 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9029672-20fe-49f6-bf3d-10d216c8acd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:30.226818 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9029672-20fe-49f6-bf3d-10d216c8acd2 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:30.227240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9029672-20fe-49f6-bf3d-10d216c8acd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:30.227653 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9029672-20fe-49f6-bf3d-10d216c8acd2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:30.302790 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9029672-20fe-49f6-bf3d-10d216c8acd2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:30.304485 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9029672-20fe-49f6-bf3d-10d216c8acd2 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:30.313933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd265528-a8ce-4ecd-a18b-514d54ece04a None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:30.315574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:30.327105 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:30.353115 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d7298e-97a1-49e8-8652-6e31126134a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:30.353983 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d7298e-97a1-49e8-8652-6e31126134a8 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:30.354297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d7298e-97a1-49e8-8652-6e31126134a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:30.354671 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d7298e-97a1-49e8-8652-6e31126134a8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:30.365270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d7298e-97a1-49e8-8652-6e31126134a8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:30.366855 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d7298e-97a1-49e8-8652-6e31126134a8 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:30.386004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-c5d7298e-97a1-49e8-8652-6e31126134a8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:30.386004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:30.386004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:30.386004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:30.386004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:30.386004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:30.386004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:30.386004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:30.386004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:30.386004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:30.386004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:30.386004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:30.386004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:30.386004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:30.386004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:30.386004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:30.387089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:30.387089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:30.387089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:30.387089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:30.387089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:30.387089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:30.387089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:30.387089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:30.387089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:30.387089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:30.387089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:30.387089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:30.387089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:30.387089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:30.387089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:30.387089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:30.387089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:30.387984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:30.387984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:30.387984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:30.387984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 337/661] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:30.389078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d7298e-97a1-49e8-8652-6e31126134a8 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:30.419808 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 325/662] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:13:30.420917 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9029672-20fe-49f6-bf3d-10d216c8acd2 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:30.504283 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d7298e-97a1-49e8-8652-6e31126134a8 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:30.524640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9029672-20fe-49f6-bf3d-10d216c8acd2 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:30.542078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89021c7-3666-437e-9448-9bc82cbb9048 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:30.542684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89021c7-3666-437e-9448-9bc82cbb9048 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:30.542963 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89021c7-3666-437e-9448-9bc82cbb9048 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:30.543231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89021c7-3666-437e-9448-9bc82cbb9048 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:30.543953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89021c7-3666-437e-9448-9bc82cbb9048 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:30.544255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89021c7-3666-437e-9448-9bc82cbb9048 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:30.545705 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89021c7-3666-437e-9448-9bc82cbb9048 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:30.547606 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:30.563163 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89c4692-2942-464f-b8f1-b45de2ad0c5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:30.563783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89c4692-2942-464f-b8f1-b45de2ad0c5b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:30.564127 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89c4692-2942-464f-b8f1-b45de2ad0c5b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:30.564463 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89c4692-2942-464f-b8f1-b45de2ad0c5b None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:30.569172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 338/663] 10.222.0.85 () {56 vars in 1139 bytes} [Tue Aug 30 04:13:30 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:30.569306 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89c4692-2942-464f-b8f1-b45de2ad0c5b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:30.569306 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89c4692-2942-464f-b8f1-b45de2ad0c5b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:30.569495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89021c7-3666-437e-9448-9bc82cbb9048 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:30.569583 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89c4692-2942-464f-b8f1-b45de2ad0c5b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:30.580176 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 326/664] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:30.581089 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89c4692-2942-464f-b8f1-b45de2ad0c5b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:30.590214 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:30.686537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89021c7-3666-437e-9448-9bc82cbb9048 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:30.696594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89c4692-2942-464f-b8f1-b45de2ad0c5b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:30.723677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de910fe-04f8-4eb6-9269-c41d29e0bf36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:30.724454 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de910fe-04f8-4eb6-9269-c41d29e0bf36 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:30.724835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de910fe-04f8-4eb6-9269-c41d29e0bf36 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:30.725258 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de910fe-04f8-4eb6-9269-c41d29e0bf36 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:30.737078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d78cc36-1c47-4947-8ac4-89d9da9edb05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:30.737078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d78cc36-1c47-4947-8ac4-89d9da9edb05 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:30.737078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d78cc36-1c47-4947-8ac4-89d9da9edb05 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:30.737078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d78cc36-1c47-4947-8ac4-89d9da9edb05 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:30.737555 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de910fe-04f8-4eb6-9269-c41d29e0bf36 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:30.738667 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de910fe-04f8-4eb6-9269-c41d29e0bf36 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:30.747045 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d78cc36-1c47-4947-8ac4-89d9da9edb05 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:30.753014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-2d78cc36-1c47-4947-8ac4-89d9da9edb05 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:30.753014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:30.753014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:30.753014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:30.753014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:30.753014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:30.753014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:30.753014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:30.753014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:30.753014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:30.753014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:30.753014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:30.753014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:30.753014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:30.753014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:30.753014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:30.754168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:30.754168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:30.754168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:30.754168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:30.754168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:30.754168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:30.754168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:30.754168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:30.754168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:30.754168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:30.754168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:30.754168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:30.754168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:30.754168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:30.754168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:30.754168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:30.755316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:30.755316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:30.755316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:30.755316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:30.755316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:30.755316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:30.755316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:30.755316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:30.755316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:30.755316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:30.755316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:30.755316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:30.756525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-3de910fe-04f8-4eb6-9269-c41d29e0bf36 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:30.756525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:30.756525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:30.756525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:30.756525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:30.756525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:30.756525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:30.756525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:30.756525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:30.756525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:30.756525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:30.756525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:30.756525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:30.756525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:30.756525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:30.756525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:30.757604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:30.757604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:30.757604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:30.757604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:30.757604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:30.757604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:30.757604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:30.757604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:30.757604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:30.757604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:30.757604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:30.757604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:30.757604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:30.757604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:30.757604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:30.757604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:30.757604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:30.758487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:30.758487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:30.758487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:30.758487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 339/665] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:30.758487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de910fe-04f8-4eb6-9269-c41d29e0bf36 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:30.761078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 327/666] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:30.761373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d78cc36-1c47-4947-8ac4-89d9da9edb05 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:30.806475 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:30.825823 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:30.868215 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de910fe-04f8-4eb6-9269-c41d29e0bf36 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:30.872816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d78cc36-1c47-4947-8ac4-89d9da9edb05 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:30.906076 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08f9452-5c32-4cbe-ac30-3a1af0880877 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:30.906674 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08f9452-5c32-4cbe-ac30-3a1af0880877 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:30.906951 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08f9452-5c32-4cbe-ac30-3a1af0880877 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:30.907217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08f9452-5c32-4cbe-ac30-3a1af0880877 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:30.908054 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3123262-7c00-437c-9245-1351821aebb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:30.908687 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3123262-7c00-437c-9245-1351821aebb7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:30.908974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3123262-7c00-437c-9245-1351821aebb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:30.909279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3123262-7c00-437c-9245-1351821aebb7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:30.916615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3123262-7c00-437c-9245-1351821aebb7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:30.918117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3123262-7c00-437c-9245-1351821aebb7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:30.930938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-a3123262-7c00-437c-9245-1351821aebb7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:30.930938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:30.930938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:30.930938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:30.930938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:30.930938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:30.930938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:30.930938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:30.930938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:30.930938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:30.930938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:30.930938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:30.930938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:30.930938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:30.930938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:30.931827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:30.931827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:30.931827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:30.931827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:30.931827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:30.931827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:30.931827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:30.931827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:30.931827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:30.931827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:30.931827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:30.931827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:30.931827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:30.931827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:30.931827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:30.931827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:30.931827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:30.932712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:30.932712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:30.932712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:30.932712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:30.932712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:30.932712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:30.932712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:30.932712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:30.933196 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 328/667] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:30.933976 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3123262-7c00-437c-9245-1351821aebb7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:30.984235 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08f9452-5c32-4cbe-ac30-3a1af0880877 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:30.985749 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08f9452-5c32-4cbe-ac30-3a1af0880877 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:31.037418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:31.041267 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3123262-7c00-437c-9245-1351821aebb7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:31.049165 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:31.077796 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd98df4-6a53-4e7c-b209-1beaeb61c4cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:31.078331 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd98df4-6a53-4e7c-b209-1beaeb61c4cd None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:31.078766 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd98df4-6a53-4e7c-b209-1beaeb61c4cd None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:31.079041 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd98df4-6a53-4e7c-b209-1beaeb61c4cd None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:31.079840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd98df4-6a53-4e7c-b209-1beaeb61c4cd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:31.080147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd98df4-6a53-4e7c-b209-1beaeb61c4cd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:31.081852 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd98df4-6a53-4e7c-b209-1beaeb61c4cd None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:31.092955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 340/668] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:13:31.092955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08f9452-5c32-4cbe-ac30-3a1af0880877 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:31.103966 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 329/669] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:31.103966 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd98df4-6a53-4e7c-b209-1beaeb61c4cd None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:31.205919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd98df4-6a53-4e7c-b209-1beaeb61c4cd None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:31.209431 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08f9452-5c32-4cbe-ac30-3a1af0880877 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:31.242562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a8846d-a345-4818-8d5d-6f0640ab76b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:31.243207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a8846d-a345-4818-8d5d-6f0640ab76b0 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:31.243584 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a8846d-a345-4818-8d5d-6f0640ab76b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:31.243963 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a8846d-a345-4818-8d5d-6f0640ab76b0 None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:31.244718 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a8846d-a345-4818-8d5d-6f0640ab76b0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:31.244911 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0441d5af-026c-4207-82db-9dc983c84d77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:31.245058 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a8846d-a345-4818-8d5d-6f0640ab76b0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:31.245551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0441d5af-026c-4207-82db-9dc983c84d77 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:31.245833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0441d5af-026c-4207-82db-9dc983c84d77 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:31.246107 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0441d5af-026c-4207-82db-9dc983c84d77 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:31.246460 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a8846d-a345-4818-8d5d-6f0640ab76b0 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:31.246768 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0441d5af-026c-4207-82db-9dc983c84d77 None admin] RBAC: Adding query filter params (name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:31.247072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0441d5af-026c-4207-82db-9dc983c84d77 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:31.248430 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0441d5af-026c-4207-82db-9dc983c84d77 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:31.261070 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 341/670] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:13:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:31.261070 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0441d5af-026c-4207-82db-9dc983c84d77 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:31.263387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 330/671] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:31.264231 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a8846d-a345-4818-8d5d-6f0640ab76b0 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:31.287832 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:31.293068 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:31.371547 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a8846d-a345-4818-8d5d-6f0640ab76b0 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:31.372546 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0441d5af-026c-4207-82db-9dc983c84d77 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:31.409112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59b5853-8e6d-4c67-bd4c-25f7a4dca0e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:31.409753 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59b5853-8e6d-4c67-bd4c-25f7a4dca0e9 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:31.410034 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59b5853-8e6d-4c67-bd4c-25f7a4dca0e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:31.410340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59b5853-8e6d-4c67-bd4c-25f7a4dca0e9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:31.410958 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e04f0b9-e706-473c-a3a1-23a7c61b5223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:31.411592 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e04f0b9-e706-473c-a3a1-23a7c61b5223 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:31.411873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e04f0b9-e706-473c-a3a1-23a7c61b5223 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:31.412143 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e04f0b9-e706-473c-a3a1-23a7c61b5223 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:31.417367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59b5853-8e6d-4c67-bd4c-25f7a4dca0e9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:31.418822 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59b5853-8e6d-4c67-bd4c-25f7a4dca0e9 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:31.423625 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e04f0b9-e706-473c-a3a1-23a7c61b5223 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:31.425071 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e04f0b9-e706-473c-a3a1-23a7c61b5223 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:31.430860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-f59b5853-8e6d-4c67-bd4c-25f7a4dca0e9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:31.430860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:31.430860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:31.430860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:31.430860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:31.430860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:31.430860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:31.430860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:31.430860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:31.430860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:31.430860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:31.430860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:31.430860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:31.430860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:31.430860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:31.431726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:31.431726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:31.431726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:31.431726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:31.431726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:31.431726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:31.431726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:31.431726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:31.431726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:31.431726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:31.431726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:31.431726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:31.431726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:31.431726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:31.431726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:31.431726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:31.431726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:31.432632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:31.432632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:31.432632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:31.432632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:31.432632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:31.432632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:31.432632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:31.432632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:31.432632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 331/672] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:31.433972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59b5853-8e6d-4c67-bd4c-25f7a4dca0e9 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:31.445299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-4e04f0b9-e706-473c-a3a1-23a7c61b5223 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:31.445299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:31.445299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:31.445299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:31.445299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:31.445299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:31.445299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:31.445299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:31.445299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:31.445299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:31.445299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:31.445299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:31.445299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:31.445299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:31.445299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:31.445299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:31.446877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:31.446877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:31.446877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:31.446877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:31.446877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:31.446877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:31.446877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:31.446877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:31.446877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:31.446877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:31.446877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:31.446877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:31.446877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:31.446877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:31.446877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:31.446877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:31.446877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:31.448012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:31.448012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:31.448012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:31.448012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 342/673] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:31.448012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e04f0b9-e706-473c-a3a1-23a7c61b5223 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:31.528187 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:31.531286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:31.550294 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59b5853-8e6d-4c67-bd4c-25f7a4dca0e9 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:31.554262 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e04f0b9-e706-473c-a3a1-23a7c61b5223 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:31.587401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66a550b-cf26-4eb7-9e6f-c2ea80d1253d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:31.588116 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66a550b-cf26-4eb7-9e6f-c2ea80d1253d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:31.588401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66a550b-cf26-4eb7-9e6f-c2ea80d1253d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:31.588674 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66a550b-cf26-4eb7-9e6f-c2ea80d1253d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:31.590435 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5412d4-149e-4499-99f0-ca31e7c9d71f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:31.591019 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5412d4-149e-4499-99f0-ca31e7c9d71f None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:31.591306 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5412d4-149e-4499-99f0-ca31e7c9d71f None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:31.591584 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5412d4-149e-4499-99f0-ca31e7c9d71f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:31.592307 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5412d4-149e-4499-99f0-ca31e7c9d71f None admin] RBAC: Adding query filter params (role.id=755eb1bfc0cf445d8f282358f404a320, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=4bc4d455ebf041d488b1199ca58358a4) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:31.592613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5412d4-149e-4499-99f0-ca31e7c9d71f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:31.594066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5412d4-149e-4499-99f0-ca31e7c9d71f None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:31.600300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66a550b-cf26-4eb7-9e6f-c2ea80d1253d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:31.600788 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 343/674] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:13:31 2022] GET /identity/v3/role_assignments?user.id=4bc4d455ebf041d488b1199ca58358a4&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=755eb1bfc0cf445d8f282358f404a320 => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:31.601624 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5412d4-149e-4499-99f0-ca31e7c9d71f None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:31.610741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66a550b-cf26-4eb7-9e6f-c2ea80d1253d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:31.634359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-f66a550b-cf26-4eb7-9e6f-c2ea80d1253d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:31.634359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:31.634359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:31.634359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:31.634359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:31.634359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:31.634359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:31.634359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:31.634359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:31.634359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:31.634359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:31.634359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:31.634359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:31.634359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:31.634359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:31.634359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:31.635367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:31.635367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:31.635367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:31.635367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:31.635367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:31.635367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:31.635367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:31.635367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:31.635367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:31.635367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:31.635367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:31.635367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:31.635367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:31.635367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:31.635367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:31.635367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:31.635367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:31.636257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:31.636257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:31.636257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:31.639446 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 332/675] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:31.640613 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66a550b-cf26-4eb7-9e6f-c2ea80d1253d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:31.744133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5412d4-149e-4499-99f0-ca31e7c9d71f None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:31.763669 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66a550b-cf26-4eb7-9e6f-c2ea80d1253d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:31.781409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d36b59a-1384-4760-9400-81336f4ec6a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:31.782021 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d36b59a-1384-4760-9400-81336f4ec6a0 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:31.782303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d36b59a-1384-4760-9400-81336f4ec6a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:31.782578 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d36b59a-1384-4760-9400-81336f4ec6a0 None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:31.783268 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d36b59a-1384-4760-9400-81336f4ec6a0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:31.783574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d36b59a-1384-4760-9400-81336f4ec6a0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:31.785073 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d36b59a-1384-4760-9400-81336f4ec6a0 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:31.795203 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:31.800295 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 344/676] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:31.805065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d36b59a-1384-4760-9400-81336f4ec6a0 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:31.805197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e06e88-e478-478d-bb99-15d783c63337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:31.805197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e06e88-e478-478d-bb99-15d783c63337 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:31.805354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:31.805425 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e06e88-e478-478d-bb99-15d783c63337 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:31.805425 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e06e88-e478-478d-bb99-15d783c63337 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:31.805425 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e06e88-e478-478d-bb99-15d783c63337 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:31.805805 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e06e88-e478-478d-bb99-15d783c63337 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:31.807449 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e06e88-e478-478d-bb99-15d783c63337 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:31.822138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 333/677] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:31.823432 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18113fa-1f7b-4e84-bdb2-4b2b22b2a578 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:31.823715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18113fa-1f7b-4e84-bdb2-4b2b22b2a578 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:31.823986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18113fa-1f7b-4e84-bdb2-4b2b22b2a578 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:31.826022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 334/678] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:31.826742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18113fa-1f7b-4e84-bdb2-4b2b22b2a578 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:31.917381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d36b59a-1384-4760-9400-81336f4ec6a0 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:31.935209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18113fa-1f7b-4e84-bdb2-4b2b22b2a578 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:31.954573 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26e4b60-0cb1-4207-baa8-b7e94479522c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:31.955180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26e4b60-0cb1-4207-baa8-b7e94479522c None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:31.955576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26e4b60-0cb1-4207-baa8-b7e94479522c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:31.955853 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26e4b60-0cb1-4207-baa8-b7e94479522c None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:31.956508 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26e4b60-0cb1-4207-baa8-b7e94479522c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:31.956804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26e4b60-0cb1-4207-baa8-b7e94479522c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:31.958979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26e4b60-0cb1-4207-baa8-b7e94479522c None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:31.970527 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae4dd8d-7d89-4948-8b65-7b777758fc98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:31.971141 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae4dd8d-7d89-4948-8b65-7b777758fc98 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:31.975881 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae4dd8d-7d89-4948-8b65-7b777758fc98 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:31.976042 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 345/679] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:31.976135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae4dd8d-7d89-4948-8b65-7b777758fc98 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:31.976135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae4dd8d-7d89-4948-8b65-7b777758fc98 None admin] RBAC: Adding query filter params (role.id=755eb1bfc0cf445d8f282358f404a320, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=7ef280ce39b14085be329f60b22530d0) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:31.976288 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26e4b60-0cb1-4207-baa8-b7e94479522c None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:31.976363 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae4dd8d-7d89-4948-8b65-7b777758fc98 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:31.976363 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae4dd8d-7d89-4948-8b65-7b777758fc98 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:31.981828 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 335/680] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:13:31 2022] GET /identity/v3/role_assignments?user.id=7ef280ce39b14085be329f60b22530d0&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=755eb1bfc0cf445d8f282358f404a320 => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:31.989169 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f11f44-f9ac-4b24-9fe3-1e57e8d77efe None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:31.989169 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f11f44-f9ac-4b24-9fe3-1e57e8d77efe None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:31.989169 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f11f44-f9ac-4b24-9fe3-1e57e8d77efe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:32.002574 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-48f11f44-f9ac-4b24-9fe3-1e57e8d77efe None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:32.003828 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-48f11f44-f9ac-4b24-9fe3-1e57e8d77efe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:32.065914 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:32.074626 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:32.097635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 336/681] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:32.098895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f11f44-f9ac-4b24-9fe3-1e57e8d77efe None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:32.127973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26e4b60-0cb1-4207-baa8-b7e94479522c None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:32.173290 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df94ccbf-c4e6-4635-ad44-8d6b95db4c8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:32.174183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df94ccbf-c4e6-4635-ad44-8d6b95db4c8e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:32.174422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df94ccbf-c4e6-4635-ad44-8d6b95db4c8e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:32.174698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df94ccbf-c4e6-4635-ad44-8d6b95db4c8e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:32.183866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df94ccbf-c4e6-4635-ad44-8d6b95db4c8e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:32.185986 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df94ccbf-c4e6-4635-ad44-8d6b95db4c8e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:32.201241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-df94ccbf-c4e6-4635-ad44-8d6b95db4c8e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:32.201241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:32.201241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:32.201241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:32.201241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:32.201241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:32.201241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:32.201241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:32.201241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:32.201241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:32.201241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:32.201241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:32.201241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:32.201241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:32.201241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:32.202465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:32.202465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:32.202465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:32.202465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:32.202465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:32.202465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:32.202465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:32.202465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:32.202465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:32.202465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:32.202465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:32.202465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:32.202465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:32.202465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:32.202465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:32.202465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:32.202465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:32.203731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:32.203731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:32.203731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:32.203731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:32.203731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:32.203731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:32.203731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:32.203731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:32.203731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 346/682] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:32.210997 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df94ccbf-c4e6-4635-ad44-8d6b95db4c8e None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:32.240376 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f11f44-f9ac-4b24-9fe3-1e57e8d77efe None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:32.277373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec10219e-cef5-494e-a8c8-3a1d0bb5920f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:32.278348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec10219e-cef5-494e-a8c8-3a1d0bb5920f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:32.278833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec10219e-cef5-494e-a8c8-3a1d0bb5920f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:32.279314 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec10219e-cef5-494e-a8c8-3a1d0bb5920f None admin] PATH_INFO: `/v3/users/swift` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:32.288221 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec10219e-cef5-494e-a8c8-3a1d0bb5920f None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:32.289972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec10219e-cef5-494e-a8c8-3a1d0bb5920f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:32.303361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-ec10219e-cef5-494e-a8c8-3a1d0bb5920f None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:13:32.303361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:32.303361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:32.303361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:32.303361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:32.303361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:32.303361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:32.303361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:32.303361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:32.303361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:32.303361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:32.303361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:32.303361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:32.303361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:32.303361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:32.303361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:32.304348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:32.304348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:32.304348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:32.304348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:32.304348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:32.304348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:32.304348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:32.304348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:32.304348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:32.304348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:32.304348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:32.304348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:32.304348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:32.304348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:32.304348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:32.304348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:32.304348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:32.304348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:32.305313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:32.305313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:32.305313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:32.305313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:32.305313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:32.305313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:32.305313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:32.305313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:32.305313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:13:32.305313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:32.307842 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 337/683] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:13:32 2022] GET /identity/v3/users/swift => generated 83 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:32.314237 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec10219e-cef5-494e-a8c8-3a1d0bb5920f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:32.318358 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df94ccbf-c4e6-4635-ad44-8d6b95db4c8e None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:32.345443 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:32.356647 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c582e2-fe1f-45cd-b12f-cf93e7b8d5b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:32.357347 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c582e2-fe1f-45cd-b12f-cf93e7b8d5b9 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:32.357634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c582e2-fe1f-45cd-b12f-cf93e7b8d5b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:32.357909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c582e2-fe1f-45cd-b12f-cf93e7b8d5b9 None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:32.358671 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c582e2-fe1f-45cd-b12f-cf93e7b8d5b9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:32.358978 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c582e2-fe1f-45cd-b12f-cf93e7b8d5b9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:32.360681 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c582e2-fe1f-45cd-b12f-cf93e7b8d5b9 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:32.368014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:32.374752 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 347/684] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:32.383323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c582e2-fe1f-45cd-b12f-cf93e7b8d5b9 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:32.436706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec10219e-cef5-494e-a8c8-3a1d0bb5920f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:32.487267 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7a8a95-4e28-4197-9a16-b29c82953f6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:32.487894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7a8a95-4e28-4197-9a16-b29c82953f6a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:32.488181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7a8a95-4e28-4197-9a16-b29c82953f6a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:32.488453 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7a8a95-4e28-4197-9a16-b29c82953f6a None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:32.489263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7a8a95-4e28-4197-9a16-b29c82953f6a None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:32.490130 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7a8a95-4e28-4197-9a16-b29c82953f6a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:32.491970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7a8a95-4e28-4197-9a16-b29c82953f6a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:32.509131 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c582e2-fe1f-45cd-b12f-cf93e7b8d5b9 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:32.521514 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 338/685] 10.222.0.85 () {56 vars in 1135 bytes} [Tue Aug 30 04:13:32 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:32.522371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7a8a95-4e28-4197-9a16-b29c82953f6a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:32.560931 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b140ea4b-0cdd-4a9a-8625-8b5df637eed8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:32.561830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b140ea4b-0cdd-4a9a-8625-8b5df637eed8 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:32.562983 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b140ea4b-0cdd-4a9a-8625-8b5df637eed8 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:32.563279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b140ea4b-0cdd-4a9a-8625-8b5df637eed8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:32.564025 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b140ea4b-0cdd-4a9a-8625-8b5df637eed8 None admin] RBAC: Adding query filter params (role.id=755eb1bfc0cf445d8f282358f404a320, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=5c2edd77896a4534888d6758ec0ee0bc) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:32.564340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b140ea4b-0cdd-4a9a-8625-8b5df637eed8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:32.565869 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b140ea4b-0cdd-4a9a-8625-8b5df637eed8 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:32.575070 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 348/686] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:13:32 2022] GET /identity/v3/role_assignments?user.id=5c2edd77896a4534888d6758ec0ee0bc&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=755eb1bfc0cf445d8f282358f404a320 => generated 608 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:32.576490 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b140ea4b-0cdd-4a9a-8625-8b5df637eed8 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:32.644541 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:32.724119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b140ea4b-0cdd-4a9a-8625-8b5df637eed8 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:32.735600 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7a8a95-4e28-4197-9a16-b29c82953f6a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:32.738333 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:32.764291 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5c788a-8514-466e-a234-e19ce82dab93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:32.764909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5c788a-8514-466e-a234-e19ce82dab93 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:32.765224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5c788a-8514-466e-a234-e19ce82dab93 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:32.765499 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5c788a-8514-466e-a234-e19ce82dab93 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:32.818186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc9edc7-5369-4b4f-b9df-a3dd901f49d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:32.819208 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc9edc7-5369-4b4f-b9df-a3dd901f49d9 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:32.819493 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc9edc7-5369-4b4f-b9df-a3dd901f49d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:32.819770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc9edc7-5369-4b4f-b9df-a3dd901f49d9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:32.835476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc9edc7-5369-4b4f-b9df-a3dd901f49d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:32.838523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc9edc7-5369-4b4f-b9df-a3dd901f49d9 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:32.852992 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5c788a-8514-466e-a234-e19ce82dab93 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:32.854573 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5c788a-8514-466e-a234-e19ce82dab93 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:32.860452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-5cc9edc7-5369-4b4f-b9df-a3dd901f49d9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:32.860452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:32.860452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:32.860452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:32.860452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:32.860452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:32.860452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:32.860452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:32.860452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:32.860452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:32.860452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:32.860452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:32.860452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:32.860452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:32.860452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:32.860452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:32.861567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:32.861567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:32.861567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:32.861567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:32.861567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:32.861567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:32.861567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:32.861567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:32.861567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:32.861567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:32.861567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:32.861567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:32.861567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:32.861567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:32.861567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:32.861567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:32.861567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:32.862557 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:32.862557 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:32.862557 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:32.866225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 339/687] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 344 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:32.873130 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc9edc7-5369-4b4f-b9df-a3dd901f49d9 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:32.916354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:32.985451 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 349/688] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:13:32.986957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5c788a-8514-466e-a234-e19ce82dab93 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:33.011236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc9edc7-5369-4b4f-b9df-a3dd901f49d9 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:33.056778 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bfeb96-468b-456d-a7ef-6d9c52aff408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:33.057476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bfeb96-468b-456d-a7ef-6d9c52aff408 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:33.057758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bfeb96-468b-456d-a7ef-6d9c52aff408 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:33.058037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bfeb96-468b-456d-a7ef-6d9c52aff408 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:33.058737 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0bfeb96-468b-456d-a7ef-6d9c52aff408 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:33.059045 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0bfeb96-468b-456d-a7ef-6d9c52aff408 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:33.060422 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0bfeb96-468b-456d-a7ef-6d9c52aff408 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:33.084996 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 340/689] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:33.090386 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bfeb96-468b-456d-a7ef-6d9c52aff408 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:33.100972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5c788a-8514-466e-a234-e19ce82dab93 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:33.139257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31dfd68-4e30-47d4-9910-988b17d5bcdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:33.140390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31dfd68-4e30-47d4-9910-988b17d5bcdf None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:33.140889 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31dfd68-4e30-47d4-9910-988b17d5bcdf None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:33.141405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31dfd68-4e30-47d4-9910-988b17d5bcdf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:33.142732 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:33.155238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31dfd68-4e30-47d4-9910-988b17d5bcdf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:33.157147 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31dfd68-4e30-47d4-9910-988b17d5bcdf None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:33.164816 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:33.175688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-e31dfd68-4e30-47d4-9910-988b17d5bcdf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:33.175688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:33.175688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:33.175688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:33.175688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:33.175688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:33.175688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:33.175688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:33.175688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:33.175688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:33.175688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:33.175688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:33.175688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:33.175688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:33.175688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:33.175688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:33.177238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:33.177238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:33.177238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:33.177238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:33.177238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:33.177238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:33.177238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:33.177238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:33.177238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:33.177238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:33.177238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:33.177238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:33.177238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:33.177238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:33.177238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:33.177238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:33.177238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:33.178150 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:33.178150 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:33.178150 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:33.180259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 350/690] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:33.185434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31dfd68-4e30-47d4-9910-988b17d5bcdf None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:33.255462 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bfeb96-468b-456d-a7ef-6d9c52aff408 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:33.291162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745a5300-ad16-4cf6-b457-98c68aa813ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:33.291748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745a5300-ad16-4cf6-b457-98c68aa813ed None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:33.291994 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745a5300-ad16-4cf6-b457-98c68aa813ed None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:33.292267 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745a5300-ad16-4cf6-b457-98c68aa813ed None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:33.292928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745a5300-ad16-4cf6-b457-98c68aa813ed None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:33.294480 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745a5300-ad16-4cf6-b457-98c68aa813ed None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:33.298000 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31dfd68-4e30-47d4-9910-988b17d5bcdf None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:33.320910 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 341/691] 10.222.0.85 () {60 vars in 1132 bytes} [Tue Aug 30 04:13:33 2022] POST /identity/v3/users => generated 254 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:13:33.352097 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140242ac-5df9-437b-8dd4-1f0eeac53de4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:33.352986 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140242ac-5df9-437b-8dd4-1f0eeac53de4 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:33.353471 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140242ac-5df9-437b-8dd4-1f0eeac53de4 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:33.353907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140242ac-5df9-437b-8dd4-1f0eeac53de4 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:33.354769 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140242ac-5df9-437b-8dd4-1f0eeac53de4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:33.355302 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140242ac-5df9-437b-8dd4-1f0eeac53de4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:33.356951 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140242ac-5df9-437b-8dd4-1f0eeac53de4 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:33.372369 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 351/692] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:33.377884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745a5300-ad16-4cf6-b457-98c68aa813ed None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:33.493507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:33.534557 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745a5300-ad16-4cf6-b457-98c68aa813ed None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:33.578891 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f72eda-6525-459c-9d90-fbdf23b8433b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:33.579539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f72eda-6525-459c-9d90-fbdf23b8433b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:33.579819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f72eda-6525-459c-9d90-fbdf23b8433b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:33.580087 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f72eda-6525-459c-9d90-fbdf23b8433b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:33.588762 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f72eda-6525-459c-9d90-fbdf23b8433b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:33.590256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f72eda-6525-459c-9d90-fbdf23b8433b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:33.602767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-53f72eda-6525-459c-9d90-fbdf23b8433b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:33.602767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:33.602767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:33.602767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:33.602767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:33.602767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:33.602767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:33.602767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:33.602767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:33.602767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:33.602767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:33.602767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:33.602767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:33.602767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:33.602767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:33.603641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:33.603641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:33.603641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:33.603641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:33.603641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:33.603641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:33.603641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:33.603641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:33.603641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:33.603641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:33.603641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:33.603641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:33.603641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:33.603641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:33.603641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:33.603641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:33.603641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:33.604551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:33.604551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:33.604551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:33.604551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:33.604551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:33.604551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:33.604551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:33.604551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:33.604997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 342/693] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:33.610109 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140242ac-5df9-437b-8dd4-1f0eeac53de4 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:33.678037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:33.722799 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140242ac-5df9-437b-8dd4-1f0eeac53de4 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:33.758480 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852d0bcb-584a-4716-8394-fe3ac32e19a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:33.759460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852d0bcb-584a-4716-8394-fe3ac32e19a9 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:33.759963 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852d0bcb-584a-4716-8394-fe3ac32e19a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:33.760437 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852d0bcb-584a-4716-8394-fe3ac32e19a9 None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:33.761360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852d0bcb-584a-4716-8394-fe3ac32e19a9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:33.761876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852d0bcb-584a-4716-8394-fe3ac32e19a9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:33.763497 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852d0bcb-584a-4716-8394-fe3ac32e19a9 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:33.776887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 352/694] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:33.783130 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f72eda-6525-459c-9d90-fbdf23b8433b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:33.808464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528b47d6-dfe4-4f7a-8c43-ebca4641401d None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:33.808976 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528b47d6-dfe4-4f7a-8c43-ebca4641401d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:33.809442 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528b47d6-dfe4-4f7a-8c43-ebca4641401d None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:33.811013 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 353/695] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:33.817578 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82876af7-e89b-4934-a883-4d941694dc9e None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:33.817938 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82876af7-e89b-4934-a883-4d941694dc9e None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:33.818210 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82876af7-e89b-4934-a883-4d941694dc9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:33.839293 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-82876af7-e89b-4934-a883-4d941694dc9e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:33.840594 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-82876af7-e89b-4934-a883-4d941694dc9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:33.894207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f72eda-6525-459c-9d90-fbdf23b8433b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:33.926351 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 354/696] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:13:33.949997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:33.955511 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc40a2a-56b1-40c3-a2d4-d79bb737a4c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:33.956338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc40a2a-56b1-40c3-a2d4-d79bb737a4c1 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:33.956579 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc40a2a-56b1-40c3-a2d4-d79bb737a4c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:33.956848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc40a2a-56b1-40c3-a2d4-d79bb737a4c1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:33.957622 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc40a2a-56b1-40c3-a2d4-d79bb737a4c1 None admin] RBAC: Adding query filter params (role.id=755eb1bfc0cf445d8f282358f404a320, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=6cb539ef6a69406baa70f660e1a815b4) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:33.958443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc40a2a-56b1-40c3-a2d4-d79bb737a4c1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:33.959877 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc40a2a-56b1-40c3-a2d4-d79bb737a4c1 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:33.972870 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 343/697] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:13:33 2022] GET /identity/v3/role_assignments?user.id=6cb539ef6a69406baa70f660e1a815b4&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=755eb1bfc0cf445d8f282358f404a320 => generated 608 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:34.231300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82876af7-e89b-4934-a883-4d941694dc9e None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:34.285549 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:34.343106 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82876af7-e89b-4934-a883-4d941694dc9e None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:34.380692 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1c8c5d-0e47-479f-9ced-c7979e56e28e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:34.381542 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1c8c5d-0e47-479f-9ced-c7979e56e28e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:34.381823 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1c8c5d-0e47-479f-9ced-c7979e56e28e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:34.382096 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1c8c5d-0e47-479f-9ced-c7979e56e28e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:34.392497 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1c8c5d-0e47-479f-9ced-c7979e56e28e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:34.397446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-9d1c8c5d-0e47-479f-9ced-c7979e56e28e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:13:34.397446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:34.397446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:34.397446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:34.397446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:34.397446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:34.397446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:34.397446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:34.397446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:34.397446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:34.397446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:34.397446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:34.397446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:34.397446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:34.397446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:34.397446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:34.398399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:34.398399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:34.398399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:34.398399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:34.398399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:34.398399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:34.398399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:34.398399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:34.398399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:34.398399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:34.398399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:34.398399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:34.398399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:34.398399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:34.398399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:34.398399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:34.399267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:34.399267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:34.399267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:34.399267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:34.399267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:34.399267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:34.399267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:34.399267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:34.399267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:34.399267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:34.399267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:13:34.399267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:34.403393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 355/698] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:13:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:34.404754 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc40a2a-56b1-40c3-a2d4-d79bb737a4c1 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:34.510717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f40e94-3cd1-4822-87ee-45a8a7dd4ab1 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:34.511074 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f40e94-3cd1-4822-87ee-45a8a7dd4ab1 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:34.511268 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f40e94-3cd1-4822-87ee-45a8a7dd4ab1 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:34.512061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc40a2a-56b1-40c3-a2d4-d79bb737a4c1 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:34.515439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 356/699] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:34.519292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9182145-0d7b-4247-87fd-43437392197d None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:34.519575 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9182145-0d7b-4247-87fd-43437392197d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:34.519970 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9182145-0d7b-4247-87fd-43437392197d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:34.542570 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-d9182145-0d7b-4247-87fd-43437392197d None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:34.543916 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d9182145-0d7b-4247-87fd-43437392197d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:34.565451 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1b2ce2-fbc1-424a-8a4a-377a19982ba5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:34.566076 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1b2ce2-fbc1-424a-8a4a-377a19982ba5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:34.566354 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1b2ce2-fbc1-424a-8a4a-377a19982ba5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:34.566635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1b2ce2-fbc1-424a-8a4a-377a19982ba5 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:34.567357 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1b2ce2-fbc1-424a-8a4a-377a19982ba5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:34.568159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1b2ce2-fbc1-424a-8a4a-377a19982ba5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:34.569818 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1b2ce2-fbc1-424a-8a4a-377a19982ba5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:34.583742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 344/700] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:13:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:34.586895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1b2ce2-fbc1-424a-8a4a-377a19982ba5 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:34.636381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 357/701] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 04:13:34.667101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:34.721525 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1b2ce2-fbc1-424a-8a4a-377a19982ba5 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:34.765068 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51b3b32-f0e0-404f-92b2-ffd180906272 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:34.765715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51b3b32-f0e0-404f-92b2-ffd180906272 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:34.765994 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51b3b32-f0e0-404f-92b2-ffd180906272 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:34.766270 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51b3b32-f0e0-404f-92b2-ffd180906272 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:34.777607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51b3b32-f0e0-404f-92b2-ffd180906272 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:34.779072 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51b3b32-f0e0-404f-92b2-ffd180906272 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:34.794210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-d51b3b32-f0e0-404f-92b2-ffd180906272 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:34.794210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:34.794210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:34.794210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:34.794210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:34.794210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:34.794210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:34.794210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:34.794210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:34.794210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:34.794210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:34.794210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:34.794210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:34.794210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:34.794210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:34.794210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:34.795179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:34.795179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:34.795179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:34.795179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:34.795179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:34.795179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:34.795179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:34.795179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:34.795179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:34.795179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:34.795179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:34.795179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:34.795179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:34.795179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:34.795179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:34.795179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:34.795179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:34.796055 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:34.796055 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:34.796055 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:34.800026 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 345/702] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:34.801415 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9182145-0d7b-4247-87fd-43437392197d None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:34.850050 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:34.928066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9182145-0d7b-4247-87fd-43437392197d None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:34.976003 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d880998e-9b57-44e5-a155-dabc400d9bef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:34.976623 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d880998e-9b57-44e5-a155-dabc400d9bef None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:34.976900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d880998e-9b57-44e5-a155-dabc400d9bef None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:34.977197 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d880998e-9b57-44e5-a155-dabc400d9bef None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:34.977850 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d880998e-9b57-44e5-a155-dabc400d9bef None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:34.978150 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d880998e-9b57-44e5-a155-dabc400d9bef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:34.979517 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d880998e-9b57-44e5-a155-dabc400d9bef None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:34.996342 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 358/703] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:35.001901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51b3b32-f0e0-404f-92b2-ffd180906272 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:35.022572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d880998e-9b57-44e5-a155-dabc400d9bef None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:35.190904 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:35.193535 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51b3b32-f0e0-404f-92b2-ffd180906272 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:35.195871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d880998e-9b57-44e5-a155-dabc400d9bef None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:35.205363 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:35.231980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3f43f6-7746-4296-a1a6-4644722ed98d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:35.232984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3f43f6-7746-4296-a1a6-4644722ed98d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:35.233708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3f43f6-7746-4296-a1a6-4644722ed98d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:35.234221 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3f43f6-7746-4296-a1a6-4644722ed98d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:35.251128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3f43f6-7746-4296-a1a6-4644722ed98d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:35.251861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5aaf14-fbfe-4a3c-93c1-f9a6319e605b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:35.252512 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5aaf14-fbfe-4a3c-93c1-f9a6319e605b None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:35.252789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5aaf14-fbfe-4a3c-93c1-f9a6319e605b None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:35.253078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5aaf14-fbfe-4a3c-93c1-f9a6319e605b None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:35.253674 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5aaf14-fbfe-4a3c-93c1-f9a6319e605b None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:35.254109 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3f43f6-7746-4296-a1a6-4644722ed98d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:35.255596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5aaf14-fbfe-4a3c-93c1-f9a6319e605b None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:35.267147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-ad3f43f6-7746-4296-a1a6-4644722ed98d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:13:35.267147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:35.267147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:35.267147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:35.267147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:35.267147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:35.267147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:35.267147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:35.267147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:35.267147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:35.267147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:35.267147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:35.267147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:35.267147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:35.267147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:35.267147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:35.268117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:35.268117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:35.268117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:35.268117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:35.268117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:35.268117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:35.268117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:35.268117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:35.268117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:35.268117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:35.268117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:35.268117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:35.268117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:35.268117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:35.268117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:35.268117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:35.268117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:35.268117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:35.269101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:35.269101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:35.269101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:35.269101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:35.269101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:35.269101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:35.269101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:35.269101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:35.269101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:13:35.269101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:35.269680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-cb5aaf14-fbfe-4a3c-93c1-f9a6319e605b None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 04:13:35.269680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:35.269680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:13:35.269680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:13:35.269680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:13:35.269680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:13:35.269680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:13:35.269680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:35.269680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 04:13:35.269680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:35.269680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:13:35.269680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:35.269680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:35.269680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:35.269680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:35.269680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:35.269680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:35.270719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:35.270719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:35.270719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:35.270719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:35.270719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:35.270719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:35.270719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:13:35.270719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:13:35.270719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:13:35.270719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:13:35.270719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:35.270719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:35.270719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:35.270719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:35.270719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:35.270719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:35.272968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:35.272968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:35.272968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:35.272968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:35.272968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:35.272968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:35.272968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:35.272968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:35.272968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:35.272968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:35.272968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:13:35.272968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:35.272968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 04:13:35.272968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:35.272968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 359/704] 10.222.0.85 () {56 vars in 1129 bytes} [Tue Aug 30 04:13:35 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:35.275494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fb9343-c071-4c13-a949-f0d665aac1cd None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:35.275780 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fb9343-c071-4c13-a949-f0d665aac1cd None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:35.276057 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fb9343-c071-4c13-a949-f0d665aac1cd None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:35.277191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 346/705] 10.222.0.85 () {56 vars in 1093 bytes} [Tue Aug 30 04:13:35 2022] GET /identity/v3/users/nova => generated 82 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:35.277675 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 360/706] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:35 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:35.278419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3f43f6-7746-4296-a1a6-4644722ed98d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:35.287777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b48f555-3f14-4457-8010-a262448781c3 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:35.288069 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b48f555-3f14-4457-8010-a262448781c3 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:35.288321 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b48f555-3f14-4457-8010-a262448781c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:35.309840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-0b48f555-3f14-4457-8010-a262448781c3 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:35.311367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-0b48f555-3f14-4457-8010-a262448781c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:35.406580 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3f43f6-7746-4296-a1a6-4644722ed98d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:35.416286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 361/707] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:35.417842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b48f555-3f14-4457-8010-a262448781c3 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:35.447578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29449f3d-ed71-4db5-8428-a12ac10988e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:35.448119 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29449f3d-ed71-4db5-8428-a12ac10988e0 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:35.448448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29449f3d-ed71-4db5-8428-a12ac10988e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:35.448727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29449f3d-ed71-4db5-8428-a12ac10988e0 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:35.449471 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29449f3d-ed71-4db5-8428-a12ac10988e0 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:35.449775 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29449f3d-ed71-4db5-8428-a12ac10988e0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:35.451171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29449f3d-ed71-4db5-8428-a12ac10988e0 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:35.456604 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 347/708] 10.222.0.85 () {56 vars in 1124 bytes} [Tue Aug 30 04:13:35 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:35.462222 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29449f3d-ed71-4db5-8428-a12ac10988e0 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:35.533432 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:35.576329 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:35.594436 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b48f555-3f14-4457-8010-a262448781c3 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:35.628461 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29449f3d-ed71-4db5-8428-a12ac10988e0 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:35.665065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f8273a-e8b6-4acc-aab8-1e1db4cf8083 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:35.665901 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f8273a-e8b6-4acc-aab8-1e1db4cf8083 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:35.666189 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f8273a-e8b6-4acc-aab8-1e1db4cf8083 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:35.666361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3be663-80e7-447e-9991-a34a5db8b30e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:35.666468 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f8273a-e8b6-4acc-aab8-1e1db4cf8083 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:35.666950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3be663-80e7-447e-9991-a34a5db8b30e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:35.667201 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f8273a-e8b6-4acc-aab8-1e1db4cf8083 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:35.667301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3be663-80e7-447e-9991-a34a5db8b30e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:35.667496 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3be663-80e7-447e-9991-a34a5db8b30e None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:35.667661 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f8273a-e8b6-4acc-aab8-1e1db4cf8083 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:35.668131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3be663-80e7-447e-9991-a34a5db8b30e None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:35.668436 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3be663-80e7-447e-9991-a34a5db8b30e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:35.668930 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f8273a-e8b6-4acc-aab8-1e1db4cf8083 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:35.669823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3be663-80e7-447e-9991-a34a5db8b30e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:35.675820 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 348/709] 10.222.0.85 () {56 vars in 1124 bytes} [Tue Aug 30 04:13:35 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:35.712935 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3be663-80e7-447e-9991-a34a5db8b30e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:35.713718 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 362/710] 10.222.0.85 () {56 vars in 1133 bytes} [Tue Aug 30 04:13:35 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:35.725065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f8273a-e8b6-4acc-aab8-1e1db4cf8083 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:35.898138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:35.903175 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3be663-80e7-447e-9991-a34a5db8b30e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:35.915845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f8273a-e8b6-4acc-aab8-1e1db4cf8083 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:35.940032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb31343-57af-42d3-96b7-1fd34dc0a158 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:35.940995 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb31343-57af-42d3-96b7-1fd34dc0a158 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:35.941527 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb31343-57af-42d3-96b7-1fd34dc0a158 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:35.942031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb31343-57af-42d3-96b7-1fd34dc0a158 None admin] PATH_INFO: `/v3/services/network` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:35.942852 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb31343-57af-42d3-96b7-1fd34dc0a158 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:35.944434 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb31343-57af-42d3-96b7-1fd34dc0a158 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:35.953978 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e803f95-8232-4d20-9dee-53e006a93f84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:35.954596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e803f95-8232-4d20-9dee-53e006a93f84 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:35.954876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e803f95-8232-4d20-9dee-53e006a93f84 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:35.955149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e803f95-8232-4d20-9dee-53e006a93f84 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:35.958532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-ceb31343-57af-42d3-96b7-1fd34dc0a158 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 04:13:35.958532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:35.958532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:13:35.958532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:13:35.958532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:13:35.958532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:13:35.958532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:13:35.958532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:35.958532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 04:13:35.958532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:35.958532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:13:35.958532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:35.958532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:35.958532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:35.958532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:35.958532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:35.958532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:35.959541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:35.959541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:35.959541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:35.959541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:35.959541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:35.959541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:35.959541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:13:35.959541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:13:35.959541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:13:35.959541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:13:35.959541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:35.959541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:35.959541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:35.959541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:35.959541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:35.959541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:35.960416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:35.960416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:35.960416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:35.960416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:35.960416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:35.960416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:35.960416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:35.960416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:35.960416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:35.960416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:35.960416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:13:35.960416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:35.960416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 04:13:35.960416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:35.967763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 349/711] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:35 2022] GET /identity/v3/services/network => generated 88 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:35.969352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb31343-57af-42d3-96b7-1fd34dc0a158 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:35.976810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:36.047961 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e803f95-8232-4d20-9dee-53e006a93f84 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:36.049467 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e803f95-8232-4d20-9dee-53e006a93f84 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:36.074810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb31343-57af-42d3-96b7-1fd34dc0a158 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:36.126784 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e353ffc2-f292-4ee2-b86d-49766524300e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:36.127817 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e353ffc2-f292-4ee2-b86d-49766524300e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:36.128304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e353ffc2-f292-4ee2-b86d-49766524300e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:36.128803 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e353ffc2-f292-4ee2-b86d-49766524300e None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:36.129967 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e353ffc2-f292-4ee2-b86d-49766524300e None admin] RBAC: Adding query filter params (name=network) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:36.130475 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e353ffc2-f292-4ee2-b86d-49766524300e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:36.132415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e353ffc2-f292-4ee2-b86d-49766524300e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:36.142204 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 350/712] 10.222.0.85 () {56 vars in 1112 bytes} [Tue Aug 30 04:13:35 2022] GET /identity/v3/services?name=network => generated 124 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:36.144189 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90993e85-e5ca-45fa-8aae-5e5b8216829c None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:36.144714 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90993e85-e5ca-45fa-8aae-5e5b8216829c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:36.145750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90993e85-e5ca-45fa-8aae-5e5b8216829c None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:36.148026 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 351/713] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:36 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:36.149310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90993e85-e5ca-45fa-8aae-5e5b8216829c None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:36.166516 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 363/714] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:13:36.168169 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24d219f-f309-4804-bd59-8c1aa30991b6 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:36.168678 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24d219f-f309-4804-bd59-8c1aa30991b6 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:36.169120 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24d219f-f309-4804-bd59-8c1aa30991b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:36.185497 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:36.191439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-e24d219f-f309-4804-bd59-8c1aa30991b6 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:36.192990 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e24d219f-f309-4804-bd59-8c1aa30991b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:36.246866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:36.270659 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90993e85-e5ca-45fa-8aae-5e5b8216829c None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:36.272414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 364/715] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:36.273080 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24d219f-f309-4804-bd59-8c1aa30991b6 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:36.311765 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cece6c-9041-4c50-bea6-bb093f3fe644 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:36.312817 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cece6c-9041-4c50-bea6-bb093f3fe644 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:36.313400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cece6c-9041-4c50-bea6-bb093f3fe644 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:36.313914 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cece6c-9041-4c50-bea6-bb093f3fe644 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:36.314755 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cece6c-9041-4c50-bea6-bb093f3fe644 None admin] RBAC: Adding query filter params (type=network) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:36.315280 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cece6c-9041-4c50-bea6-bb093f3fe644 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:36.317620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cece6c-9041-4c50-bea6-bb093f3fe644 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:36.329231 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 352/716] 10.222.0.85 () {56 vars in 1112 bytes} [Tue Aug 30 04:13:36 2022] GET /identity/v3/services?type=network => generated 124 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:36.388961 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24d219f-f309-4804-bd59-8c1aa30991b6 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:36.431496 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d401f7eb-4a97-4471-8168-a0c117c014fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:36.432151 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d401f7eb-4a97-4471-8168-a0c117c014fc None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:36.432493 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d401f7eb-4a97-4471-8168-a0c117c014fc None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:36.432781 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d401f7eb-4a97-4471-8168-a0c117c014fc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:36.445165 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d401f7eb-4a97-4471-8168-a0c117c014fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:36.446815 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d401f7eb-4a97-4471-8168-a0c117c014fc None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:36.463868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-d401f7eb-4a97-4471-8168-a0c117c014fc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:36.463868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:36.463868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:36.463868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:36.463868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:36.463868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:36.463868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:36.463868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:36.463868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:36.463868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:36.463868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:36.463868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:36.463868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:36.463868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:36.463868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:36.463868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:36.464925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:36.464925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:36.464925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:36.464925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:36.464925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:36.464925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:36.464925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:36.464925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:36.464925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:36.464925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:36.464925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:36.464925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:36.464925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:36.464925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:36.464925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:36.464925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:36.464925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:36.466486 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:36.466486 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:36.466486 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:36.466486 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 365/717] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:36.470988 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cece6c-9041-4c50-bea6-bb093f3fe644 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:36.581503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:36.593407 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cece6c-9041-4c50-bea6-bb093f3fe644 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:36.631703 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f469cf-181f-4a49-8221-ee3c4fec4ec2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:36.632328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f469cf-181f-4a49-8221-ee3c4fec4ec2 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:36.632610 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f469cf-181f-4a49-8221-ee3c4fec4ec2 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:36.632876 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f469cf-181f-4a49-8221-ee3c4fec4ec2 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:36.633558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f469cf-181f-4a49-8221-ee3c4fec4ec2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:36.633851 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f469cf-181f-4a49-8221-ee3c4fec4ec2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:36.635332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f469cf-181f-4a49-8221-ee3c4fec4ec2 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:36.649502 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 353/718] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:36.656160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d401f7eb-4a97-4471-8168-a0c117c014fc None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:36.674405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f469cf-181f-4a49-8221-ee3c4fec4ec2 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:36.721065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:36.833322 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d401f7eb-4a97-4471-8168-a0c117c014fc None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:36.844146 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f469cf-181f-4a49-8221-ee3c4fec4ec2 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:36.876246 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad252f59-ed5b-4b2e-805e-e00e77b8aa13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:36.876894 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad252f59-ed5b-4b2e-805e-e00e77b8aa13 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:36.877216 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad252f59-ed5b-4b2e-805e-e00e77b8aa13 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:36.877494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad252f59-ed5b-4b2e-805e-e00e77b8aa13 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:36.880332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062f1876-deda-4809-b9ce-4ab23ab7c9f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:36.880972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062f1876-deda-4809-b9ce-4ab23ab7c9f1 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:36.881530 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062f1876-deda-4809-b9ce-4ab23ab7c9f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:36.881801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062f1876-deda-4809-b9ce-4ab23ab7c9f1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:36.885372 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad252f59-ed5b-4b2e-805e-e00e77b8aa13 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:36.886845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad252f59-ed5b-4b2e-805e-e00e77b8aa13 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:36.892468 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062f1876-deda-4809-b9ce-4ab23ab7c9f1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:36.898387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-062f1876-deda-4809-b9ce-4ab23ab7c9f1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:36.898387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:36.898387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:36.898387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:36.898387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:36.898387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:36.898387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:36.898387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:36.898387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:36.898387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:36.898387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:36.898387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:36.898387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:36.898387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:36.898387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:36.898387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:36.899304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:36.899304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:36.899304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:36.899304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:36.899304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:36.899304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:36.899304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:36.899304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:36.899304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:36.899304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:36.899304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:36.899304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:36.899304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:36.899304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:36.899304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:36.899304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:36.900701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:36.900701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:36.900701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:36.900701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:36.900701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:36.900701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:36.900701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:36.900701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:36.900701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:36.900701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:36.900701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:36.900701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:36.901622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-ad252f59-ed5b-4b2e-805e-e00e77b8aa13 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:36.901622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:36.901622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:36.901622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:36.901622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:36.901622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:36.901622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:36.901622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:36.901622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:36.901622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:36.901622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:36.901622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:36.901622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:36.901622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:36.901622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:36.903412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:36.903412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:36.903412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:36.903412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:36.903412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:36.903412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:36.903412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:36.903412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:36.903412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:36.903412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:36.903412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:36.903412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:36.903412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:36.903412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:36.903412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:36.903412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:36.903412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:36.906746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:36.906746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:36.906746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:36.906746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:36.906746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:36.906746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:36.906746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:36.906746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:36.907740 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 354/719] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:36 2022] GET /identity/v3/roles/service => generated 85 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:36.907883 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 366/720] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:36.907883 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad252f59-ed5b-4b2e-805e-e00e77b8aa13 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:36.908148 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2c9d8e-47c3-450b-8a77-9a8acdd3cc2c None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:36.908148 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2c9d8e-47c3-450b-8a77-9a8acdd3cc2c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:36.908148 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2c9d8e-47c3-450b-8a77-9a8acdd3cc2c None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:36.913979 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 355/721] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:36 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:36.913979 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2c9d8e-47c3-450b-8a77-9a8acdd3cc2c None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:36.931344 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:37.014171 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:37.023795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad252f59-ed5b-4b2e-805e-e00e77b8aa13 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:37.035275 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2c9d8e-47c3-450b-8a77-9a8acdd3cc2c None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:37.061996 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17295de-a614-4112-8cb3-d983f3e5f22f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:37.062847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17295de-a614-4112-8cb3-d983f3e5f22f None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:37.063198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17295de-a614-4112-8cb3-d983f3e5f22f None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:37.063564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17295de-a614-4112-8cb3-d983f3e5f22f None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:37.064313 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17295de-a614-4112-8cb3-d983f3e5f22f None admin] RBAC: Adding query filter params (name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:37.064686 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17295de-a614-4112-8cb3-d983f3e5f22f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:37.066357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17295de-a614-4112-8cb3-d983f3e5f22f None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:37.076760 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467c72fe-68eb-41df-9c89-82181e475c4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:37.077352 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 367/722] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:13:36 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:37.078222 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467c72fe-68eb-41df-9c89-82181e475c4e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:37.078699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467c72fe-68eb-41df-9c89-82181e475c4e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:37.078806 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b815d548-8127-435b-9b94-576e99225946 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:37.079166 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b815d548-8127-435b-9b94-576e99225946 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:37.079438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467c72fe-68eb-41df-9c89-82181e475c4e None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:37.079549 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b815d548-8127-435b-9b94-576e99225946 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:37.080423 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467c72fe-68eb-41df-9c89-82181e475c4e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:37.080912 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467c72fe-68eb-41df-9c89-82181e475c4e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:37.082687 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467c72fe-68eb-41df-9c89-82181e475c4e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:37.101272 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 356/723] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:37.101439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-b815d548-8127-435b-9b94-576e99225946 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:37.102585 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467c72fe-68eb-41df-9c89-82181e475c4e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:37.102697 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b815d548-8127-435b-9b94-576e99225946 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:37.197448 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 368/724] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:37.198491 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b815d548-8127-435b-9b94-576e99225946 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:37.206568 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:37.244649 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467c72fe-68eb-41df-9c89-82181e475c4e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:37.292612 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d04dba7-e4bd-4491-b830-4e1f2401af31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:37.293664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d04dba7-e4bd-4491-b830-4e1f2401af31 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:37.295732 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d04dba7-e4bd-4491-b830-4e1f2401af31 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:37.296202 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d04dba7-e4bd-4491-b830-4e1f2401af31 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:37.321289 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d04dba7-e4bd-4491-b830-4e1f2401af31 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:37.324638 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d04dba7-e4bd-4491-b830-4e1f2401af31 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:37.362997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-3d04dba7-e4bd-4491-b830-4e1f2401af31 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:37.362997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:37.362997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:37.362997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:37.362997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:37.362997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:37.362997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:37.362997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:37.362997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:37.362997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:37.362997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:37.362997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:37.362997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:37.362997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:37.362997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:37.362997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:37.364061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:37.364061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:37.364061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:37.364061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:37.364061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:37.364061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:37.364061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:37.364061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:37.364061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:37.364061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:37.364061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:37.364061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:37.364061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:37.364061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:37.364061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:37.364061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:37.364061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:37.364997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:37.364997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:37.364997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:37.366969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 357/725] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:37.372119 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d04dba7-e4bd-4491-b830-4e1f2401af31 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:37.387550 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:37.407147 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b815d548-8127-435b-9b94-576e99225946 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:37.445107 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee78d72-7b18-4f5c-8cc0-291a83265e7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:37.446004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee78d72-7b18-4f5c-8cc0-291a83265e7d None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:37.446496 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee78d72-7b18-4f5c-8cc0-291a83265e7d None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:37.446920 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee78d72-7b18-4f5c-8cc0-291a83265e7d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:37.447820 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee78d72-7b18-4f5c-8cc0-291a83265e7d None admin] RBAC: Adding query filter params (role.id=d17954e7ce544f0d90a512340232c60e, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=4bc4d455ebf041d488b1199ca58358a4) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:37.448337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee78d72-7b18-4f5c-8cc0-291a83265e7d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:37.451005 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee78d72-7b18-4f5c-8cc0-291a83265e7d None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:37.463524 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 369/726] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:13:37 2022] GET /identity/v3/role_assignments?user.id=4bc4d455ebf041d488b1199ca58358a4&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=d17954e7ce544f0d90a512340232c60e => generated 259 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:37.517143 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee78d72-7b18-4f5c-8cc0-291a83265e7d None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:37.564350 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d04dba7-e4bd-4491-b830-4e1f2401af31 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:37.591784 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:37.604989 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e466f9fc-90cf-4e98-b066-621967f5dd73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:37.605863 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e466f9fc-90cf-4e98-b066-621967f5dd73 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:37.606216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e466f9fc-90cf-4e98-b066-621967f5dd73 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:37.606560 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e466f9fc-90cf-4e98-b066-621967f5dd73 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:37.607318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e466f9fc-90cf-4e98-b066-621967f5dd73 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:37.607693 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e466f9fc-90cf-4e98-b066-621967f5dd73 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:37.609233 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e466f9fc-90cf-4e98-b066-621967f5dd73 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:37.626605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 358/727] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:37.635467 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e466f9fc-90cf-4e98-b066-621967f5dd73 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:37.713718 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee78d72-7b18-4f5c-8cc0-291a83265e7d None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:37.755979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eecbbc8-042d-4ecb-a936-9b9c2b822fc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:37.756879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eecbbc8-042d-4ecb-a936-9b9c2b822fc7 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:37.757171 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eecbbc8-042d-4ecb-a936-9b9c2b822fc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:37.757444 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eecbbc8-042d-4ecb-a936-9b9c2b822fc7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:37.769880 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eecbbc8-042d-4ecb-a936-9b9c2b822fc7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:37.776004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-8eecbbc8-042d-4ecb-a936-9b9c2b822fc7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:13:37.776004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:37.776004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:37.776004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:37.776004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:37.776004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:37.776004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:37.776004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:37.776004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:37.776004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:37.776004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:37.776004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:37.776004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:37.776004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:37.776004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:37.776004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:37.776941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:37.776941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:37.776941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:37.776941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:37.776941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:37.776941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:37.776941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:37.776941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:37.776941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:37.776941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:37.776941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:37.776941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:37.776941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:37.776941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:37.776941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:37.776941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:37.777808 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:37.777808 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:37.777808 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:37.777808 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:37.777808 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:37.777808 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:37.777808 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:37.777808 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:37.777808 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:37.777808 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:37.777808 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:13:37.777808 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:37.777808 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 370/728] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:13:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:37.783010 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eecbbc8-042d-4ecb-a936-9b9c2b822fc7 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:37.805412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:37.855137 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e466f9fc-90cf-4e98-b066-621967f5dd73 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:37.927374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df083161-b4a3-449a-a528-bac04cf1ca25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:37.927932 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df083161-b4a3-449a-a528-bac04cf1ca25 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:37.928211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df083161-b4a3-449a-a528-bac04cf1ca25 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:37.928487 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df083161-b4a3-449a-a528-bac04cf1ca25 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:37.945567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df083161-b4a3-449a-a528-bac04cf1ca25 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:37.947054 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df083161-b4a3-449a-a528-bac04cf1ca25 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:37.970571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-df083161-b4a3-449a-a528-bac04cf1ca25 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 04:13:37.970571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:37.970571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:37.970571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:37.970571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:37.970571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:37.970571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:37.970571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:37.970571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:37.970571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:37.970571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:37.970571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:37.970571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:37.970571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:37.970571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:37.970571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:37.971741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:37.971741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:37.971741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:37.971741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:37.971741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:37.971741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:37.971741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:37.971741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:37.971741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:37.971741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:37.971741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:37.971741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:37.971741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:37.971741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:37.971741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:37.971741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:37.971741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:37.971741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:37.972716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:37.972716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:37.972716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:37.972716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:37.972716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:37.972716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:37.972716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:37.972716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:37.972716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 04:13:37.972716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:37.973313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 359/729] 10.222.0.85 () {56 vars in 1117 bytes} [Tue Aug 30 04:13:37 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 338 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:37.982666 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df083161-b4a3-449a-a528-bac04cf1ca25 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:38.002593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eecbbc8-042d-4ecb-a936-9b9c2b822fc7 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:38.070776 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:38.078660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26872fef-40ef-49de-956d-eefeb766fb01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:38.079308 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26872fef-40ef-49de-956d-eefeb766fb01 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:38.079593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26872fef-40ef-49de-956d-eefeb766fb01 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:38.079867 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26872fef-40ef-49de-956d-eefeb766fb01 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:38.080563 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26872fef-40ef-49de-956d-eefeb766fb01 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:38.080867 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26872fef-40ef-49de-956d-eefeb766fb01 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:38.083747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26872fef-40ef-49de-956d-eefeb766fb01 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:38.093335 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 371/730] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:13:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:38.102201 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26872fef-40ef-49de-956d-eefeb766fb01 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:38.152559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df083161-b4a3-449a-a528-bac04cf1ca25 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:38.190555 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1b73f6-087a-45b4-8419-ef1ac1c505a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:38.191338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1b73f6-087a-45b4-8419-ef1ac1c505a3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:38.191714 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1b73f6-087a-45b4-8419-ef1ac1c505a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:38.192238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1b73f6-087a-45b4-8419-ef1ac1c505a3 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:38.192802 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:38.193218 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1b73f6-087a-45b4-8419-ef1ac1c505a3 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:38.193620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1b73f6-087a-45b4-8419-ef1ac1c505a3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:38.195155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1b73f6-087a-45b4-8419-ef1ac1c505a3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:38.224105 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 360/731] 10.222.0.85 () {56 vars in 1149 bytes} [Tue Aug 30 04:13:37 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:38.224754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26872fef-40ef-49de-956d-eefeb766fb01 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:38.225846 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324dfa8c-c276-4406-8ec0-4144a64be369 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:38.226226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324dfa8c-c276-4406-8ec0-4144a64be369 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:38.226666 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324dfa8c-c276-4406-8ec0-4144a64be369 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:38.228469 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 361/732] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:38.229305 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324dfa8c-c276-4406-8ec0-4144a64be369 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:38.268486 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a74c7f-6e9d-4aea-b627-3dca5e3ee761 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:38.269154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a74c7f-6e9d-4aea-b627-3dca5e3ee761 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:38.270799 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a74c7f-6e9d-4aea-b627-3dca5e3ee761 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:38.271083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a74c7f-6e9d-4aea-b627-3dca5e3ee761 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:38.285257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a74c7f-6e9d-4aea-b627-3dca5e3ee761 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:38.287586 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a74c7f-6e9d-4aea-b627-3dca5e3ee761 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:38.307089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-e3a74c7f-6e9d-4aea-b627-3dca5e3ee761 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:38.307089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:38.307089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:38.307089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:38.307089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:38.307089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:38.307089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:38.307089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:38.307089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:38.307089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:38.307089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:38.307089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:38.307089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:38.307089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:38.307089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:38.307089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:38.308124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:38.308124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:38.308124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:38.308124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:38.308124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:38.308124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:38.308124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:38.308124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:38.308124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:38.308124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:38.308124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:38.308124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:38.308124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:38.308124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:38.308124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:38.308124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:38.308124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:38.309290 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:38.309290 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:38.309290 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:38.309290 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 372/733] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:38.312782 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b33da9-0467-498f-8138-531c6be18b7c None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:38.312782 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b33da9-0467-498f-8138-531c6be18b7c None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:38.312782 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b33da9-0467-498f-8138-531c6be18b7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:38.335236 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-a8b33da9-0467-498f-8138-531c6be18b7c None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:38.336960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-a8b33da9-0467-498f-8138-531c6be18b7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:38.342558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:38.348156 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324dfa8c-c276-4406-8ec0-4144a64be369 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:38.387692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f130940f-2567-4047-8e39-7ffa249e9733 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:38.388466 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f130940f-2567-4047-8e39-7ffa249e9733 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:38.388839 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f130940f-2567-4047-8e39-7ffa249e9733 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:38.389323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f130940f-2567-4047-8e39-7ffa249e9733 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:38.424313 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 373/734] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:38.425461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b33da9-0467-498f-8138-531c6be18b7c None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:38.486615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f130940f-2567-4047-8e39-7ffa249e9733 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:38.488242 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f130940f-2567-4047-8e39-7ffa249e9733 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:38.522101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:38.594115 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b33da9-0467-498f-8138-531c6be18b7c None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:38.661992 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745d5f82-bfa3-4c71-8229-98b3aa0fe642 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:38.662670 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745d5f82-bfa3-4c71-8229-98b3aa0fe642 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:38.662955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745d5f82-bfa3-4c71-8229-98b3aa0fe642 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:38.663228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745d5f82-bfa3-4c71-8229-98b3aa0fe642 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:38.664225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745d5f82-bfa3-4c71-8229-98b3aa0fe642 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:38.664532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745d5f82-bfa3-4c71-8229-98b3aa0fe642 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:38.666281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745d5f82-bfa3-4c71-8229-98b3aa0fe642 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:38.667630 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:38.676996 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 362/735] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 448 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 04:13:38.678656 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfcb63a-6731-4f4b-ae22-af3b40beb61b None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:38.679171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfcb63a-6731-4f4b-ae22-af3b40beb61b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:38.679663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfcb63a-6731-4f4b-ae22-af3b40beb61b None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:38.681468 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 363/736] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:38.682413 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfcb63a-6731-4f4b-ae22-af3b40beb61b None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:38.692973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 374/737] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:38.694229 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700fdde0-fd8f-4564-a431-9a2b625ff1d2 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:38.694520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700fdde0-fd8f-4564-a431-9a2b625ff1d2 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:38.694792 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700fdde0-fd8f-4564-a431-9a2b625ff1d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:38.722753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-700fdde0-fd8f-4564-a431-9a2b625ff1d2 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:38.724785 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-700fdde0-fd8f-4564-a431-9a2b625ff1d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:38.800592 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfcb63a-6731-4f4b-ae22-af3b40beb61b None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:38.806288 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 375/738] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 04:13:38.814252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700fdde0-fd8f-4564-a431-9a2b625ff1d2 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:38.848383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facf6529-7509-4700-895d-aeead6b6d86a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:38.849025 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facf6529-7509-4700-895d-aeead6b6d86a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:38.849330 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facf6529-7509-4700-895d-aeead6b6d86a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:38.849604 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facf6529-7509-4700-895d-aeead6b6d86a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:38.863325 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facf6529-7509-4700-895d-aeead6b6d86a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:38.865283 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:38.865472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facf6529-7509-4700-895d-aeead6b6d86a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:38.886304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-facf6529-7509-4700-895d-aeead6b6d86a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:38.886304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:38.886304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:38.886304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:38.886304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:38.886304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:38.886304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:38.886304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:38.886304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:38.886304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:38.886304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:38.886304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:38.886304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:38.886304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:38.886304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:38.886304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:38.887332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:38.887332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:38.887332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:38.887332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:38.887332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:38.887332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:38.887332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:38.887332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:38.887332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:38.887332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:38.887332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:38.887332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:38.887332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:38.887332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:38.887332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:38.887332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:38.887332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:38.888821 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:38.888821 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:38.888821 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:38.888821 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 364/739] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:38.888821 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facf6529-7509-4700-895d-aeead6b6d86a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:38.933805 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700fdde0-fd8f-4564-a431-9a2b625ff1d2 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:38.956338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:38.987357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead4a146-8643-415b-ac78-e859fb2b5345 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:38.987996 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead4a146-8643-415b-ac78-e859fb2b5345 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:38.988273 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead4a146-8643-415b-ac78-e859fb2b5345 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:38.988545 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead4a146-8643-415b-ac78-e859fb2b5345 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:38.997026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead4a146-8643-415b-ac78-e859fb2b5345 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:38.998544 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead4a146-8643-415b-ac78-e859fb2b5345 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:39.012119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-ead4a146-8643-415b-ac78-e859fb2b5345 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:13:39.012119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:39.012119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:39.012119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:39.012119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:39.012119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:39.012119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:39.012119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:39.012119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:39.012119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:39.012119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:39.012119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:39.012119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:39.012119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:39.012119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:39.012119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:39.013118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:39.013118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:39.013118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:39.013118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:39.013118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:39.013118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:39.013118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:39.013118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:39.013118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:39.013118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:39.013118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:39.013118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:39.013118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:39.013118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:39.013118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:39.013118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:39.013118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:39.013118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:39.014043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:39.014043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:39.014043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:39.014043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:39.014043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:39.014043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:39.014043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:39.014043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:39.014043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:13:39.014043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:39.014043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 376/740] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:13:38 2022] GET /identity/v3/users/swift => generated 83 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:39.014637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead4a146-8643-415b-ac78-e859fb2b5345 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:39.019161 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facf6529-7509-4700-895d-aeead6b6d86a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:39.064977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c79d4a6-9363-4f2e-9383-20f48cb9a315 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:39.066507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c79d4a6-9363-4f2e-9383-20f48cb9a315 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:39.067130 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c79d4a6-9363-4f2e-9383-20f48cb9a315 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:39.067610 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c79d4a6-9363-4f2e-9383-20f48cb9a315 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:39.068611 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c79d4a6-9363-4f2e-9383-20f48cb9a315 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:39.070524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c79d4a6-9363-4f2e-9383-20f48cb9a315 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:39.092488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 365/741] 10.222.0.85 () {60 vars in 1141 bytes} [Tue Aug 30 04:13:38 2022] POST /identity/v3/services => generated 251 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:13:39.093493 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c79d4a6-9363-4f2e-9383-20f48cb9a315 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:39.150476 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead4a146-8643-415b-ac78-e859fb2b5345 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:39.152420 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:39.206751 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba56aeba-5796-4ac0-9d15-b7ec37d330d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:39.207443 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba56aeba-5796-4ac0-9d15-b7ec37d330d6 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:39.207756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba56aeba-5796-4ac0-9d15-b7ec37d330d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:39.208070 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba56aeba-5796-4ac0-9d15-b7ec37d330d6 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:39.208741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba56aeba-5796-4ac0-9d15-b7ec37d330d6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:39.209055 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba56aeba-5796-4ac0-9d15-b7ec37d330d6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:39.210569 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba56aeba-5796-4ac0-9d15-b7ec37d330d6 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:39.224872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 377/742] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:39.225866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba56aeba-5796-4ac0-9d15-b7ec37d330d6 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:39.261967 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c79d4a6-9363-4f2e-9383-20f48cb9a315 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:39.276199 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:39.309542 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da68ca2-708c-41f5-9d6b-32ad0c0a8d5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:39.309883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da68ca2-708c-41f5-9d6b-32ad0c0a8d5f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:39.310157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da68ca2-708c-41f5-9d6b-32ad0c0a8d5f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:39.310427 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da68ca2-708c-41f5-9d6b-32ad0c0a8d5f None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:39.311170 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da68ca2-708c-41f5-9d6b-32ad0c0a8d5f None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:39.311466 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da68ca2-708c-41f5-9d6b-32ad0c0a8d5f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:39.312900 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da68ca2-708c-41f5-9d6b-32ad0c0a8d5f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:39.333894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 366/743] 10.222.0.85 () {56 vars in 1135 bytes} [Tue Aug 30 04:13:39 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:39.334779 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da68ca2-708c-41f5-9d6b-32ad0c0a8d5f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:39.363263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba56aeba-5796-4ac0-9d15-b7ec37d330d6 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:39.414404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61226c46-b1fe-45ee-80b6-5d3799656aca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:39.415017 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61226c46-b1fe-45ee-80b6-5d3799656aca None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:39.415564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61226c46-b1fe-45ee-80b6-5d3799656aca None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:39.415837 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61226c46-b1fe-45ee-80b6-5d3799656aca None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:39.417145 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61226c46-b1fe-45ee-80b6-5d3799656aca None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:39.418907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61226c46-b1fe-45ee-80b6-5d3799656aca None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:39.448674 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 378/744] 10.222.0.85 () {60 vars in 1141 bytes} [Tue Aug 30 04:13:39 2022] POST /identity/v3/services => generated 240 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:13:39.449745 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61226c46-b1fe-45ee-80b6-5d3799656aca None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:39.475194 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:39.477439 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da68ca2-708c-41f5-9d6b-32ad0c0a8d5f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:39.573981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43295993-8d57-4c88-af14-6dda1ab8df28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:39.574644 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43295993-8d57-4c88-af14-6dda1ab8df28 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:39.574919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43295993-8d57-4c88-af14-6dda1ab8df28 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:39.575186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43295993-8d57-4c88-af14-6dda1ab8df28 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:39.590747 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43295993-8d57-4c88-af14-6dda1ab8df28 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:39.591148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61226c46-b1fe-45ee-80b6-5d3799656aca None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:39.592145 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43295993-8d57-4c88-af14-6dda1ab8df28 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:39.618594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-43295993-8d57-4c88-af14-6dda1ab8df28 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:39.618594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:39.618594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:39.618594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:39.618594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:39.618594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:39.618594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:39.618594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:39.618594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:39.618594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:39.618594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:39.618594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:39.618594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:39.618594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:39.618594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:39.619536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:39.619536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:39.619536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:39.619536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:39.619536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:39.619536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:39.619536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:39.619536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:39.619536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:39.619536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:39.619536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:39.619536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:39.619536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:39.619536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:39.619536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:39.619536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:39.619536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:39.620663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:39.620663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:39.620663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:39.620663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:39.620663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:39.620663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:39.620663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:39.620663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:39.620663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 367/745] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:39.630444 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43295993-8d57-4c88-af14-6dda1ab8df28 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:39.632922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28dd583-bd93-46b3-a358-42e1d998b58d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:39.633774 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28dd583-bd93-46b3-a358-42e1d998b58d None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:39.634213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28dd583-bd93-46b3-a358-42e1d998b58d None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:39.634668 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28dd583-bd93-46b3-a358-42e1d998b58d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:39.689823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:39.732938 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28dd583-bd93-46b3-a358-42e1d998b58d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:39.734826 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28dd583-bd93-46b3-a358-42e1d998b58d None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:39.764246 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:39.795401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43295993-8d57-4c88-af14-6dda1ab8df28 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:39.839176 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a93a71f-c423-47aa-b5bc-398cc996064e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:39.840043 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a93a71f-c423-47aa-b5bc-398cc996064e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:39.840636 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a93a71f-c423-47aa-b5bc-398cc996064e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:39.841145 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a93a71f-c423-47aa-b5bc-398cc996064e None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:39.842459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a93a71f-c423-47aa-b5bc-398cc996064e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:39.843204 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a93a71f-c423-47aa-b5bc-398cc996064e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:39.845059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a93a71f-c423-47aa-b5bc-398cc996064e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:39.861894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 368/746] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:39.863436 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61598d5-e776-433c-ba3f-160fbf1c772d None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:39.863671 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 379/747] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:39 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 04:13:39.864966 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61598d5-e776-433c-ba3f-160fbf1c772d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:39.865273 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61598d5-e776-433c-ba3f-160fbf1c772d None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:39.866735 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 369/748] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:39.868852 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28dd583-bd93-46b3-a358-42e1d998b58d None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:39.871522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61598d5-e776-433c-ba3f-160fbf1c772d None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:40.001480 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:40.005675 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61598d5-e776-433c-ba3f-160fbf1c772d None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:40.046539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28dd583-bd93-46b3-a358-42e1d998b58d None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:40.069191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:40.087671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e0d04c-0895-4c3e-ab5c-517d43f4619b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:40.088474 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e0d04c-0895-4c3e-ab5c-517d43f4619b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:40.088868 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e0d04c-0895-4c3e-ab5c-517d43f4619b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:40.089191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e0d04c-0895-4c3e-ab5c-517d43f4619b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:40.093620 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635ff419-eb75-49b2-9a98-f42c2910219b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:40.094534 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635ff419-eb75-49b2-9a98-f42c2910219b None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:40.095015 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635ff419-eb75-49b2-9a98-f42c2910219b None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:40.095506 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635ff419-eb75-49b2-9a98-f42c2910219b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:40.096949 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635ff419-eb75-49b2-9a98-f42c2910219b None admin] RBAC: Adding query filter params (role.id=755eb1bfc0cf445d8f282358f404a320, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=00aeab772672403984bdcb8baaa1520e) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:40.097577 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635ff419-eb75-49b2-9a98-f42c2910219b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:40.100325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635ff419-eb75-49b2-9a98-f42c2910219b None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:40.101450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e0d04c-0895-4c3e-ab5c-517d43f4619b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:40.103088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e0d04c-0895-4c3e-ab5c-517d43f4619b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:40.112432 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 380/749] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:13:39 2022] GET /identity/v3/role_assignments?user.id=00aeab772672403984bdcb8baaa1520e&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=755eb1bfc0cf445d8f282358f404a320 => generated 259 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:40.114277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb61b99-fd58-4914-9c30-67a32ed6fd22 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:40.114798 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb61b99-fd58-4914-9c30-67a32ed6fd22 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:40.115293 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb61b99-fd58-4914-9c30-67a32ed6fd22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:40.130578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-f0e0d04c-0895-4c3e-ab5c-517d43f4619b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:40.130578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:40.130578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:40.130578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:40.130578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:40.130578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:40.130578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:40.130578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:40.130578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:40.130578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:40.130578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:40.130578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:40.130578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:40.130578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:40.130578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:40.130578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:40.131847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:40.131847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:40.131847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:40.131847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:40.131847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:40.131847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:40.131847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:40.131847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:40.131847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:40.131847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:40.131847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:40.131847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:40.131847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:40.131847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:40.131847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:40.131847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:40.131847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:40.132976 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:40.132976 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:40.132976 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:40.132976 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 370/750] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:40.146493 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e0d04c-0895-4c3e-ab5c-517d43f4619b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:40.159092 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-6eb61b99-fd58-4914-9c30-67a32ed6fd22 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:40.160151 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-6eb61b99-fd58-4914-9c30-67a32ed6fd22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:40.320268 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 381/751] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:40 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:13:40.358988 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e0d04c-0895-4c3e-ab5c-517d43f4619b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:40.402245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a4ec06-aa32-4b26-9f0e-87ff082120fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:40.403194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a4ec06-aa32-4b26-9f0e-87ff082120fa None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:40.403482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a4ec06-aa32-4b26-9f0e-87ff082120fa None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:40.403764 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a4ec06-aa32-4b26-9f0e-87ff082120fa None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:40.404421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a4ec06-aa32-4b26-9f0e-87ff082120fa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:40.404728 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a4ec06-aa32-4b26-9f0e-87ff082120fa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:40.406413 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a4ec06-aa32-4b26-9f0e-87ff082120fa None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:40.415545 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:40.427492 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 371/752] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:40.434179 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb61b99-fd58-4914-9c30-67a32ed6fd22 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:40.567984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb61b99-fd58-4914-9c30-67a32ed6fd22 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:40.582511 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:40.608324 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb78e94c-0fc5-4d03-bda0-0552b6bf3b12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:40.609449 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb78e94c-0fc5-4d03-bda0-0552b6bf3b12 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:40.609964 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb78e94c-0fc5-4d03-bda0-0552b6bf3b12 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:40.610435 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb78e94c-0fc5-4d03-bda0-0552b6bf3b12 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:40.617994 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb78e94c-0fc5-4d03-bda0-0552b6bf3b12 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:40.619936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb78e94c-0fc5-4d03-bda0-0552b6bf3b12 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:40.632871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-cb78e94c-0fc5-4d03-bda0-0552b6bf3b12 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:40.632871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:40.632871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:40.632871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:40.632871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:40.632871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:40.632871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:40.632871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:40.632871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:40.632871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:40.632871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:40.632871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:40.632871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:40.632871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:40.632871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:40.633841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:40.633841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:40.633841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:40.633841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:40.633841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:40.633841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:40.633841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:40.633841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:40.633841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:40.633841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:40.633841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:40.633841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:40.633841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:40.633841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:40.633841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:40.633841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:40.633841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:40.634748 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:40.634748 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:40.634748 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:40.634748 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:40.634748 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:40.634748 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:40.634748 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:40.634748 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:40.637263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 382/753] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:40.646635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a4ec06-aa32-4b26-9f0e-87ff082120fa None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:40.755121 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb78e94c-0fc5-4d03-bda0-0552b6bf3b12 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:40.839968 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a4ec06-aa32-4b26-9f0e-87ff082120fa None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:40.873941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb78e94c-0fc5-4d03-bda0-0552b6bf3b12 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:40.876490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a4e579-c64c-4e19-ad3a-ededc7e0c47b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:40.877247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a4e579-c64c-4e19-ad3a-ededc7e0c47b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:40.878365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a4e579-c64c-4e19-ad3a-ededc7e0c47b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:40.878365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a4e579-c64c-4e19-ad3a-ededc7e0c47b None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:40.878850 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a4e579-c64c-4e19-ad3a-ededc7e0c47b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:40.879259 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a4e579-c64c-4e19-ad3a-ededc7e0c47b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:40.881623 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a4e579-c64c-4e19-ad3a-ededc7e0c47b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:40.899720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 372/754] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:40.901638 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a4e579-c64c-4e19-ad3a-ededc7e0c47b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:40.945581 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37af9d60-3704-49d8-8528-167dc90b0804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:40.946224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37af9d60-3704-49d8-8528-167dc90b0804 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:40.946556 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37af9d60-3704-49d8-8528-167dc90b0804 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:40.946835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37af9d60-3704-49d8-8528-167dc90b0804 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:40.970110 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37af9d60-3704-49d8-8528-167dc90b0804 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:40.981767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-37af9d60-3704-49d8-8528-167dc90b0804 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:13:40.981767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:40.981767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:40.981767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:40.981767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:40.981767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:40.981767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:40.981767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:40.981767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:40.981767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:40.981767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:40.981767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:40.981767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:40.981767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:40.981767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:40.981767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:40.982810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:40.982810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:40.982810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:40.982810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:40.982810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:40.982810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:40.982810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:40.982810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:40.982810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:40.982810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:40.982810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:40.982810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:40.982810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:40.982810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:40.982810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:40.982810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:40.983654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:40.983654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:40.983654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:40.983654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:40.983654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:40.983654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:40.983654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:40.983654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:40.983654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:40.983654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:40.983654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:13:40.983654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:40.986487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 383/755] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:13:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:40.992075 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37af9d60-3704-49d8-8528-167dc90b0804 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:41.011031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:41.072789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:41.092599 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a4e579-c64c-4e19-ad3a-ededc7e0c47b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:41.170651 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271fed66-684d-4793-9e30-7e21ea3ba537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:41.171190 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271fed66-684d-4793-9e30-7e21ea3ba537 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:41.171531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271fed66-684d-4793-9e30-7e21ea3ba537 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:41.171811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271fed66-684d-4793-9e30-7e21ea3ba537 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:41.172547 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271fed66-684d-4793-9e30-7e21ea3ba537 None admin] RBAC: Adding query filter params (role.id=d17954e7ce544f0d90a512340232c60e, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=6cb539ef6a69406baa70f660e1a815b4) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:41.172857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271fed66-684d-4793-9e30-7e21ea3ba537 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:41.174355 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271fed66-684d-4793-9e30-7e21ea3ba537 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:41.188002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 373/756] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:13:40 2022] GET /identity/v3/role_assignments?user.id=6cb539ef6a69406baa70f660e1a815b4&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=d17954e7ce544f0d90a512340232c60e => generated 259 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:41.201814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37af9d60-3704-49d8-8528-167dc90b0804 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:41.240710 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1026aadd-cf7a-4a61-a3be-a43b8e12fd54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:41.241379 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1026aadd-cf7a-4a61-a3be-a43b8e12fd54 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:41.241661 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1026aadd-cf7a-4a61-a3be-a43b8e12fd54 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:41.242007 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1026aadd-cf7a-4a61-a3be-a43b8e12fd54 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:41.242700 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1026aadd-cf7a-4a61-a3be-a43b8e12fd54 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:41.243000 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1026aadd-cf7a-4a61-a3be-a43b8e12fd54 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:41.244399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1026aadd-cf7a-4a61-a3be-a43b8e12fd54 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:41.254673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 384/757] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:13:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:41.262309 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271fed66-684d-4793-9e30-7e21ea3ba537 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:41.396783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271fed66-684d-4793-9e30-7e21ea3ba537 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:41.421864 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:41.435138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4de2fb-3e64-4007-8080-50b95b44ac76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:41.435778 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4de2fb-3e64-4007-8080-50b95b44ac76 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:41.436061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4de2fb-3e64-4007-8080-50b95b44ac76 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:41.436337 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4de2fb-3e64-4007-8080-50b95b44ac76 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:41.448983 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4de2fb-3e64-4007-8080-50b95b44ac76 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:41.450506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4de2fb-3e64-4007-8080-50b95b44ac76 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:41.466191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-da4de2fb-3e64-4007-8080-50b95b44ac76 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:41.466191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:41.466191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:41.466191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:41.466191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:41.466191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:41.466191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:41.466191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:41.466191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:41.466191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:41.466191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:41.466191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:41.466191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:41.466191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:41.466191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:41.466191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:41.467210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:41.467210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:41.467210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:41.467210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:41.467210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:41.467210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:41.467210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:41.467210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:41.467210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:41.467210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:41.467210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:41.467210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:41.467210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:41.467210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:41.467210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:41.467210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:41.467210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:41.468296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:41.468296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:41.468296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:41.468296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 374/758] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:41.474893 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1026aadd-cf7a-4a61-a3be-a43b8e12fd54 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:41.615514 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:41.625494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1026aadd-cf7a-4a61-a3be-a43b8e12fd54 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:41.678910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286e435b-0d6f-4f04-bf5c-f0b3820e37cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:41.679404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286e435b-0d6f-4f04-bf5c-f0b3820e37cb None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:41.679692 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286e435b-0d6f-4f04-bf5c-f0b3820e37cb None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:41.679964 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286e435b-0d6f-4f04-bf5c-f0b3820e37cb None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:41.680627 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286e435b-0d6f-4f04-bf5c-f0b3820e37cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:41.680933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286e435b-0d6f-4f04-bf5c-f0b3820e37cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:41.681136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f9e8bb-4b27-4d98-b6b1-f0500f9d1090 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:41.681405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f9e8bb-4b27-4d98-b6b1-f0500f9d1090 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:41.681688 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f9e8bb-4b27-4d98-b6b1-f0500f9d1090 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:41.682354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286e435b-0d6f-4f04-bf5c-f0b3820e37cb None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:41.683348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 375/759] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:41.693683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3c5dfa-70de-48c1-a6f9-07bb45c4194e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:41.693972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3c5dfa-70de-48c1-a6f9-07bb45c4194e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:41.694244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3c5dfa-70de-48c1-a6f9-07bb45c4194e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:41.701074 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 385/760] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:41.706691 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286e435b-0d6f-4f04-bf5c-f0b3820e37cb None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:41.718253 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4d3c5dfa-70de-48c1-a6f9-07bb45c4194e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:41.719461 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4d3c5dfa-70de-48c1-a6f9-07bb45c4194e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:41.807168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 376/761] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:41 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:13:41.839348 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286e435b-0d6f-4f04-bf5c-f0b3820e37cb None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:41.887645 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461c5f50-9e62-4ea6-b5de-9858df4d3238 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:41.888291 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461c5f50-9e62-4ea6-b5de-9858df4d3238 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:41.888554 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461c5f50-9e62-4ea6-b5de-9858df4d3238 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:41.888830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461c5f50-9e62-4ea6-b5de-9858df4d3238 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:41.901683 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461c5f50-9e62-4ea6-b5de-9858df4d3238 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:41.903159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461c5f50-9e62-4ea6-b5de-9858df4d3238 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:41.921465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-461c5f50-9e62-4ea6-b5de-9858df4d3238 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:13:41.921465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:41.921465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:41.921465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:41.921465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:41.921465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:41.921465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:41.921465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:41.921465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:41.921465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:41.921465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:41.921465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:41.921465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:41.921465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:41.921465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:41.921465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:41.922489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:41.922489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:41.922489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:41.922489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:41.922489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:41.922489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:41.922489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:41.922489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:41.922489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:41.922489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:41.922489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:41.922489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:41.922489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:41.922489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:41.922489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:41.922489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:41.922489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:41.922489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:41.924684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:41.924684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:41.924684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:41.924684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:41.924684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:41.924684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:41.924684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:41.924684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:41.924684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:13:41.924684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:41.924684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 386/762] 10.222.0.85 () {56 vars in 1093 bytes} [Tue Aug 30 04:13:41 2022] GET /identity/v3/users/nova => generated 82 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:41.932528 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3c5dfa-70de-48c1-a6f9-07bb45c4194e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:42.004160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266b58e4-4180-4bd7-b14b-17f435614645 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:42.004558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266b58e4-4180-4bd7-b14b-17f435614645 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:42.004790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266b58e4-4180-4bd7-b14b-17f435614645 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:42.006683 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 387/763] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:42.013014 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fb265c-48ae-447b-8221-58cde0177351 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:42.013362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fb265c-48ae-447b-8221-58cde0177351 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:42.013598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fb265c-48ae-447b-8221-58cde0177351 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:42.042932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-08fb265c-48ae-447b-8221-58cde0177351 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:42.044144 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-08fb265c-48ae-447b-8221-58cde0177351 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:42.067594 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:42.112341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3c5dfa-70de-48c1-a6f9-07bb45c4194e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:42.124692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:42.174705 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 388/764] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:42 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:13:42.181811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fb0bd6-5709-4229-8e55-5a2891cdb6b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:42.182450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fb0bd6-5709-4229-8e55-5a2891cdb6b7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:42.182796 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fb0bd6-5709-4229-8e55-5a2891cdb6b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:42.183005 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fb0bd6-5709-4229-8e55-5a2891cdb6b7 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:42.183758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fb0bd6-5709-4229-8e55-5a2891cdb6b7 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:42.184063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fb0bd6-5709-4229-8e55-5a2891cdb6b7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:42.185529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fb0bd6-5709-4229-8e55-5a2891cdb6b7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:42.214029 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 377/765] 10.222.0.85 () {56 vars in 1133 bytes} [Tue Aug 30 04:13:41 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:42.222159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fb265c-48ae-447b-8221-58cde0177351 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:42.230087 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fb0bd6-5709-4229-8e55-5a2891cdb6b7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:42.341339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fb0bd6-5709-4229-8e55-5a2891cdb6b7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:42.378756 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ff73da-f97a-4ccd-bd52-2b77c5587953 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:42.379400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ff73da-f97a-4ccd-bd52-2b77c5587953 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:42.380216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ff73da-f97a-4ccd-bd52-2b77c5587953 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:42.380493 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ff73da-f97a-4ccd-bd52-2b77c5587953 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:42.381453 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ff73da-f97a-4ccd-bd52-2b77c5587953 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:42.382948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ff73da-f97a-4ccd-bd52-2b77c5587953 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:42.392853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-e4ff73da-f97a-4ccd-bd52-2b77c5587953 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 04:13:42.392853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:42.392853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:13:42.392853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:13:42.392853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:13:42.392853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:13:42.392853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:13:42.392853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:42.392853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 04:13:42.392853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:42.392853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:13:42.392853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:42.392853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:42.392853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:42.392853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:42.392853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:42.392853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:42.394030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:42.394030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:42.394030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:42.394030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:42.394030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:42.394030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:42.394030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:13:42.394030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:13:42.394030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:13:42.394030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:13:42.394030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:42.394030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:42.394030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:42.394030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:42.394030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:42.394030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:42.395819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:42.395819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:42.395819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:42.395819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:42.395819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:42.395819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:42.395819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:42.395819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:42.395819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:42.395819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:42.395819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:13:42.395819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:42.395819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 04:13:42.395819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:42.395819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 378/766] 10.222.0.85 () {56 vars in 1129 bytes} [Tue Aug 30 04:13:42 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:42.400745 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ff73da-f97a-4ccd-bd52-2b77c5587953 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:42.430769 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:42.479653 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fb265c-48ae-447b-8221-58cde0177351 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:42.514291 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ff73da-f97a-4ccd-bd52-2b77c5587953 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:42.527434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e718bc9-6cb6-4261-b231-95e3101bd118 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:42.528088 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e718bc9-6cb6-4261-b231-95e3101bd118 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:42.528522 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e718bc9-6cb6-4261-b231-95e3101bd118 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:42.528744 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e718bc9-6cb6-4261-b231-95e3101bd118 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:42.540164 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e718bc9-6cb6-4261-b231-95e3101bd118 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:42.541875 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e718bc9-6cb6-4261-b231-95e3101bd118 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:42.545205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:42.549980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a766fd-237c-4c0e-9978-54bbf045ecc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:42.550602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a766fd-237c-4c0e-9978-54bbf045ecc6 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:42.550885 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a766fd-237c-4c0e-9978-54bbf045ecc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:42.551158 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a766fd-237c-4c0e-9978-54bbf045ecc6 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:42.552258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a766fd-237c-4c0e-9978-54bbf045ecc6 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:42.552258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a766fd-237c-4c0e-9978-54bbf045ecc6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:42.553569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a766fd-237c-4c0e-9978-54bbf045ecc6 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:42.559012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 379/767] 10.222.0.85 () {56 vars in 1124 bytes} [Tue Aug 30 04:13:42 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:42.559667 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a766fd-237c-4c0e-9978-54bbf045ecc6 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:42.565217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-1e718bc9-6cb6-4261-b231-95e3101bd118 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:42.565217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:42.565217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:42.565217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:42.565217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:42.565217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:42.565217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:42.565217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:42.565217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:42.565217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:42.565217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:42.565217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:42.565217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:42.565217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:42.565217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:42.565217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:42.566203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:42.566203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:42.566203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:42.566203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:42.566203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:42.566203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:42.566203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:42.566203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:42.566203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:42.566203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:42.566203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:42.566203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:42.566203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:42.566203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:42.566203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:42.566203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:42.566203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:42.567650 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:42.567650 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:42.567650 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:42.569213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 389/768] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 348 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:42.570581 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0165a97-9170-419c-8422-1332af120fdd None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:42.570950 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0165a97-9170-419c-8422-1332af120fdd None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:42.571282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0165a97-9170-419c-8422-1332af120fdd None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:42.576518 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 390/769] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:42.576518 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0165a97-9170-419c-8422-1332af120fdd None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:42.681534 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:42.700880 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a766fd-237c-4c0e-9978-54bbf045ecc6 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:42.701851 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0165a97-9170-419c-8422-1332af120fdd None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:42.745208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68de0c70-cb59-45e3-9eee-61a0e843eb8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:42.745929 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68de0c70-cb59-45e3-9eee-61a0e843eb8e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:42.746465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68de0c70-cb59-45e3-9eee-61a0e843eb8e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:42.746974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68de0c70-cb59-45e3-9eee-61a0e843eb8e None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:42.748149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68de0c70-cb59-45e3-9eee-61a0e843eb8e None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:42.748628 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68de0c70-cb59-45e3-9eee-61a0e843eb8e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:42.750475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68de0c70-cb59-45e3-9eee-61a0e843eb8e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:42.758148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 391/770] 10.222.0.85 () {56 vars in 1124 bytes} [Tue Aug 30 04:13:42 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:42.758941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68de0c70-cb59-45e3-9eee-61a0e843eb8e None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:42.764786 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9933ce7e-66f4-4ac8-ae7c-a3bfa52555d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:42.765801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9933ce7e-66f4-4ac8-ae7c-a3bfa52555d5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:42.766282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9933ce7e-66f4-4ac8-ae7c-a3bfa52555d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:42.766733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9933ce7e-66f4-4ac8-ae7c-a3bfa52555d5 None admin] PATH_INFO: `/v3/services/network` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:42.767514 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9933ce7e-66f4-4ac8-ae7c-a3bfa52555d5 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:42.769133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9933ce7e-66f4-4ac8-ae7c-a3bfa52555d5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:42.780084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-9933ce7e-66f4-4ac8-ae7c-a3bfa52555d5 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 04:13:42.780084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:42.780084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:13:42.780084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:13:42.780084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:13:42.780084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:13:42.780084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:13:42.780084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:42.780084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 04:13:42.780084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:42.780084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:13:42.780084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:42.780084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:42.780084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:42.780084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:42.780084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:42.780084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:42.781133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:42.781133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:42.781133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:42.781133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:42.781133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:42.781133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:42.781133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:13:42.781133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:13:42.781133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:13:42.781133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:13:42.781133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:42.781133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:42.781133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:42.781133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:42.781133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:42.781133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:42.781972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:42.781972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:42.781972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:42.781972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:42.781972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:42.781972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:42.781972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:42.781972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:42.781972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:42.781972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:42.781972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:13:42.781972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:42.781972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 04:13:42.781972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:42.786873 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 380/771] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:42 2022] GET /identity/v3/services/network => generated 88 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:42.787927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f030c48-f63d-4ede-9238-bde08cf1d2cd None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:42.788216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f030c48-f63d-4ede-9238-bde08cf1d2cd None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:42.788491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f030c48-f63d-4ede-9238-bde08cf1d2cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:42.806111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5f030c48-f63d-4ede-9238-bde08cf1d2cd None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:42.807394 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5f030c48-f63d-4ede-9238-bde08cf1d2cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:42.839353 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:42.883949 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 381/772] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:42 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:13:42.884932 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f030c48-f63d-4ede-9238-bde08cf1d2cd None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:42.889708 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68de0c70-cb59-45e3-9eee-61a0e843eb8e None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:42.933740 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370e7383-887f-4817-a6de-2075e2f2d8de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:42.934710 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370e7383-887f-4817-a6de-2075e2f2d8de None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:42.935344 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370e7383-887f-4817-a6de-2075e2f2d8de None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:42.935924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370e7383-887f-4817-a6de-2075e2f2d8de None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:42.936779 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370e7383-887f-4817-a6de-2075e2f2d8de None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:42.937352 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370e7383-887f-4817-a6de-2075e2f2d8de None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:42.939087 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370e7383-887f-4817-a6de-2075e2f2d8de None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:42.955315 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 392/773] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:42.956601 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370e7383-887f-4817-a6de-2075e2f2d8de None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:43.008307 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:43.030217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f030c48-f63d-4ede-9238-bde08cf1d2cd None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:43.076146 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370e7383-887f-4817-a6de-2075e2f2d8de None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:43.077300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fd4d87-e7ae-4412-9260-8653ebd4c1d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:43.077928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fd4d87-e7ae-4412-9260-8653ebd4c1d7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:43.078211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fd4d87-e7ae-4412-9260-8653ebd4c1d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:43.078486 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fd4d87-e7ae-4412-9260-8653ebd4c1d7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:43.079473 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10fd4d87-e7ae-4412-9260-8653ebd4c1d7 None admin] RBAC: Adding query filter params (interface=public, service_id=a963e2684a774a6bb9dd1d4db32eba0f, region_id=RegionOne) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:43.079788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10fd4d87-e7ae-4412-9260-8653ebd4c1d7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:43.082163 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10fd4d87-e7ae-4412-9260-8653ebd4c1d7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:43.097204 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 382/774] 10.222.0.85 () {56 vars in 1251 bytes} [Tue Aug 30 04:13:42 2022] GET /identity/v3/endpoints?service_id=a963e2684a774a6bb9dd1d4db32eba0f&interface=public®ion_id=RegionOne => generated 194 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:43.098541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fd4d87-e7ae-4412-9260-8653ebd4c1d7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:43.112491 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:43.133821 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907d529f-e8dd-4460-996c-f584abb28a1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:43.134852 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907d529f-e8dd-4460-996c-f584abb28a1d None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:43.135389 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907d529f-e8dd-4460-996c-f584abb28a1d None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:43.135867 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907d529f-e8dd-4460-996c-f584abb28a1d None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:43.136713 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907d529f-e8dd-4460-996c-f584abb28a1d None admin] RBAC: Adding query filter params (name=network) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:43.137282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907d529f-e8dd-4460-996c-f584abb28a1d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:43.139129 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907d529f-e8dd-4460-996c-f584abb28a1d None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:43.145854 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 393/775] 10.222.0.85 () {56 vars in 1112 bytes} [Tue Aug 30 04:13:42 2022] GET /identity/v3/services?name=network => generated 124 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:43.147105 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907d529f-e8dd-4460-996c-f584abb28a1d None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:43.228058 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fd4d87-e7ae-4412-9260-8653ebd4c1d7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:43.259455 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907d529f-e8dd-4460-996c-f584abb28a1d None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:43.268578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc79551-cfe7-45a0-a31d-6464869dc2ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:43.269275 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc79551-cfe7-45a0-a31d-6464869dc2ff None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:43.269567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc79551-cfe7-45a0-a31d-6464869dc2ff None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:43.269843 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc79551-cfe7-45a0-a31d-6464869dc2ff None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:43.278067 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc79551-cfe7-45a0-a31d-6464869dc2ff None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:43.279549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc79551-cfe7-45a0-a31d-6464869dc2ff None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:43.291689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-3bc79551-cfe7-45a0-a31d-6464869dc2ff None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:43.291689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:43.291689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:43.291689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:43.291689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:43.291689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:43.291689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:43.291689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:43.291689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:43.291689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:43.291689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:43.291689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:43.291689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:43.291689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:43.291689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:43.292777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:43.292777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:43.292777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:43.292777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:43.292777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:43.292777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:43.292777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:43.292777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:43.292777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:43.292777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:43.292777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:43.292777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:43.292777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:43.292777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:43.292777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:43.292777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:43.292777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:43.293696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:43.293696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:43.293696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:43.293696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:43.293696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:43.293696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:43.293696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:43.293696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:43.293696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 383/776] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:43.293696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc79551-cfe7-45a0-a31d-6464869dc2ff None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:43.302321 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59caaae4-9b1d-42db-ab87-3a68970aaa52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:43.302760 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59caaae4-9b1d-42db-ab87-3a68970aaa52 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:43.303043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59caaae4-9b1d-42db-ab87-3a68970aaa52 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:43.303307 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59caaae4-9b1d-42db-ab87-3a68970aaa52 None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:43.303924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59caaae4-9b1d-42db-ab87-3a68970aaa52 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:43.305416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59caaae4-9b1d-42db-ab87-3a68970aaa52 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:43.308012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:43.318379 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 394/777] 10.222.0.85 () {56 vars in 1087 bytes} [Tue Aug 30 04:13:43 2022] GET /identity/v3/services => generated 772 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:43.318379 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59caaae4-9b1d-42db-ab87-3a68970aaa52 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:43.387196 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:43.434152 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc79551-cfe7-45a0-a31d-6464869dc2ff None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:43.457490 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59caaae4-9b1d-42db-ab87-3a68970aaa52 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:43.476486 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9807a2-ea80-4933-a890-63042eae97e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:43.477159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9807a2-ea80-4933-a890-63042eae97e9 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:43.477445 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9807a2-ea80-4933-a890-63042eae97e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:43.477718 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9807a2-ea80-4933-a890-63042eae97e9 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:43.478373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9807a2-ea80-4933-a890-63042eae97e9 None admin] RBAC: Adding query filter params (type=network) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:43.478675 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9807a2-ea80-4933-a890-63042eae97e9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:43.480305 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9807a2-ea80-4933-a890-63042eae97e9 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:43.486593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 384/778] 10.222.0.85 () {56 vars in 1112 bytes} [Tue Aug 30 04:13:43 2022] GET /identity/v3/services?type=network => generated 350 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:43.487443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9807a2-ea80-4933-a890-63042eae97e9 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:43.499308 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578c45f3-91a9-44d6-a3ba-e9def43f9787 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:43.499944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578c45f3-91a9-44d6-a3ba-e9def43f9787 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:43.500221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578c45f3-91a9-44d6-a3ba-e9def43f9787 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:43.500489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578c45f3-91a9-44d6-a3ba-e9def43f9787 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:43.513139 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578c45f3-91a9-44d6-a3ba-e9def43f9787 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:43.522325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-578c45f3-91a9-44d6-a3ba-e9def43f9787 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:43.522325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:43.522325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:43.522325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:43.522325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:43.522325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:43.522325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:43.522325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:43.522325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:43.522325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:43.522325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:43.522325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:43.522325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:43.522325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:43.522325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:43.522325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:43.523319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:43.523319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:43.523319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:43.523319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:43.523319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:43.523319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:43.523319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:43.523319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:43.523319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:43.523319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:43.523319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:43.523319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:43.523319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:43.523319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:43.523319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:43.523319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:43.524160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:43.524160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:43.524160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:43.524160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:43.524160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:43.524160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:43.524160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:43.524160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:43.524160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:43.524160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:43.524160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:43.524160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:43.528144 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 395/779] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:43 2022] GET /identity/v3/roles/service => generated 85 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:43.529317 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578c45f3-91a9-44d6-a3ba-e9def43f9787 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:43.576581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:43.609566 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9807a2-ea80-4933-a890-63042eae97e9 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:43.657264 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578c45f3-91a9-44d6-a3ba-e9def43f9787 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:43.657784 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5286cd70-e783-4c7b-a589-c609a508c8f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:43.658423 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5286cd70-e783-4c7b-a589-c609a508c8f4 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:43.658906 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5286cd70-e783-4c7b-a589-c609a508c8f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:43.659196 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5286cd70-e783-4c7b-a589-c609a508c8f4 None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:43.660263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5286cd70-e783-4c7b-a589-c609a508c8f4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:43.660441 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5286cd70-e783-4c7b-a589-c609a508c8f4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:43.662238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5286cd70-e783-4c7b-a589-c609a508c8f4 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:43.668164 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:43.683540 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 385/780] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:43.683540 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421f2606-4e07-4378-8727-0787229e37ee None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:43.683540 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421f2606-4e07-4378-8727-0787229e37ee None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:43.683540 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421f2606-4e07-4378-8727-0787229e37ee None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:43.683540 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 386/781] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:43.683540 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421f2606-4e07-4378-8727-0787229e37ee None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:43.705453 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa10fd7-f584-4b47-85a0-afe348e245d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:43.706080 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa10fd7-f584-4b47-85a0-afe348e245d1 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:43.706362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa10fd7-f584-4b47-85a0-afe348e245d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:43.706638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa10fd7-f584-4b47-85a0-afe348e245d1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:43.707357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa10fd7-f584-4b47-85a0-afe348e245d1 None admin] RBAC: Adding query filter params (interface=public, service_id=5c2b2f2b327d4639818beb15698f27e7, region_id=RegionOne) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:43.707761 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa10fd7-f584-4b47-85a0-afe348e245d1 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:43.709182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa10fd7-f584-4b47-85a0-afe348e245d1 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:43.721450 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 396/782] 10.222.0.85 () {56 vars in 1251 bytes} [Tue Aug 30 04:13:43 2022] GET /identity/v3/endpoints?service_id=5c2b2f2b327d4639818beb15698f27e7&interface=public®ion_id=RegionOne => generated 194 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:43.721450 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa10fd7-f584-4b47-85a0-afe348e245d1 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:43.803198 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421f2606-4e07-4378-8727-0787229e37ee None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:43.843581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:43.851141 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2de882-ffed-4b73-9af7-fb9014fc7e28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:43.851785 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2de882-ffed-4b73-9af7-fb9014fc7e28 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:43.852063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2de882-ffed-4b73-9af7-fb9014fc7e28 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:43.852336 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2de882-ffed-4b73-9af7-fb9014fc7e28 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:43.853026 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2de882-ffed-4b73-9af7-fb9014fc7e28 None admin] RBAC: Adding query filter params (name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:43.853426 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2de882-ffed-4b73-9af7-fb9014fc7e28 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:43.854858 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2de882-ffed-4b73-9af7-fb9014fc7e28 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:43.869827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 387/783] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:13:43 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:43.869827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf05a10-2a10-4164-8090-ed0bca9c9931 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:43.869827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf05a10-2a10-4164-8090-ed0bca9c9931 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:43.869827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf05a10-2a10-4164-8090-ed0bca9c9931 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:43.889018 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-8bf05a10-2a10-4164-8090-ed0bca9c9931 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:43.890334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8bf05a10-2a10-4164-8090-ed0bca9c9931 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:43.949782 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa10fd7-f584-4b47-85a0-afe348e245d1 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:43.984296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 388/784] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:43 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:13:43.985540 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf05a10-2a10-4164-8090-ed0bca9c9931 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:43.992286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c0f5aa-501b-40f5-9bc2-b8c12c7fb607 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:43.992972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c0f5aa-501b-40f5-9bc2-b8c12c7fb607 None admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:43.993292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c0f5aa-501b-40f5-9bc2-b8c12c7fb607 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:43.993569 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c0f5aa-501b-40f5-9bc2-b8c12c7fb607 None admin] PATH_INFO: `/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/4bc4d455ebf041d488b1199ca58358a4/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:43.995490 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c0f5aa-501b-40f5-9bc2-b8c12c7fb607 None admin] RBAC: Authorizing `identity:create_grant(project_id=a860dcc3b2d64d1491c40710ec7423a5, user_id=4bc4d455ebf041d488b1199ca58358a4, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:43.997649 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c0f5aa-501b-40f5-9bc2-b8c12c7fb607 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:44.025286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 397/785] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:13:43 2022] PUT /identity/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/4bc4d455ebf041d488b1199ca58358a4/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:13:44.027260 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c0f5aa-501b-40f5-9bc2-b8c12c7fb607 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:44.095747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:44.184324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf05a10-2a10-4164-8090-ed0bca9c9931 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:44.188736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c0f5aa-501b-40f5-9bc2-b8c12c7fb607 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:44.228342 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:44.231292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843d99f2-c1e0-4229-bb4a-a249de94205e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:44.232775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843d99f2-c1e0-4229-bb4a-a249de94205e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:44.233502 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843d99f2-c1e0-4229-bb4a-a249de94205e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:44.233987 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843d99f2-c1e0-4229-bb4a-a249de94205e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:44.240723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da7594b-871f-479c-9721-df80ac53b944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:44.248610 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da7594b-871f-479c-9721-df80ac53b944 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:44.248610 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da7594b-871f-479c-9721-df80ac53b944 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:44.248610 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da7594b-871f-479c-9721-df80ac53b944 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:44.248610 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da7594b-871f-479c-9721-df80ac53b944 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:44.248610 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da7594b-871f-479c-9721-df80ac53b944 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:44.252090 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 389/786] 10.222.0.85 () {56 vars in 1087 bytes} [Tue Aug 30 04:13:43 2022] GET /identity/v3/services => generated 772 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:44.256598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843d99f2-c1e0-4229-bb4a-a249de94205e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:44.258105 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843d99f2-c1e0-4229-bb4a-a249de94205e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:44.274720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da7594b-871f-479c-9721-df80ac53b944 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:44.277818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-843d99f2-c1e0-4229-bb4a-a249de94205e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:44.277818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:44.277818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:44.277818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:44.277818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:44.277818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:44.277818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:44.277818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:44.277818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:44.277818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:44.277818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:44.277818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:44.277818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:44.277818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:44.277818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:44.277818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:44.278955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:44.278955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:44.278955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:44.278955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:44.278955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:44.278955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:44.278955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:44.278955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:44.278955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:44.278955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:44.278955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:44.278955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:44.278955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:44.278955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:44.278955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:44.278955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:44.278955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:44.280064 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:44.280064 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:44.280064 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:44.281957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 398/787] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:44.291443 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843d99f2-c1e0-4229-bb4a-a249de94205e None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:44.407628 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:44.429007 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843d99f2-c1e0-4229-bb4a-a249de94205e None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:44.472181 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a3a6a7-0703-4369-b380-8e98600cb509 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:44.472801 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a3a6a7-0703-4369-b380-8e98600cb509 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:44.473755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a3a6a7-0703-4369-b380-8e98600cb509 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:44.473755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a3a6a7-0703-4369-b380-8e98600cb509 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:44.474093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a3a6a7-0703-4369-b380-8e98600cb509 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:44.474399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a3a6a7-0703-4369-b380-8e98600cb509 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:44.476392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a3a6a7-0703-4369-b380-8e98600cb509 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:44.488394 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da7594b-871f-479c-9721-df80ac53b944 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:44.490445 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 399/788] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:44.495643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a3a6a7-0703-4369-b380-8e98600cb509 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:44.526893 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c956d5da-e068-4079-8dfa-da13aa3f8fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:44.528209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c956d5da-e068-4079-8dfa-da13aa3f8fb9 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:44.528699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c956d5da-e068-4079-8dfa-da13aa3f8fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:44.529208 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c956d5da-e068-4079-8dfa-da13aa3f8fb9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:44.542209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c956d5da-e068-4079-8dfa-da13aa3f8fb9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:44.548341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-c956d5da-e068-4079-8dfa-da13aa3f8fb9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:13:44.548341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:44.548341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:44.548341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:44.548341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:44.548341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:44.548341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:44.548341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:44.548341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:44.548341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:44.548341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:44.548341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:44.548341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:44.548341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:44.548341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:44.548341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:44.549339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:44.549339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:44.549339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:44.549339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:44.549339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:44.549339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:44.549339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:44.549339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:44.549339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:44.549339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:44.549339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:44.549339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:44.549339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:44.549339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:44.549339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:44.549339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:44.550256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:44.550256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:44.550256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:44.550256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:44.550256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:44.550256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:44.550256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:44.550256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:44.550256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:44.550256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:44.550256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:13:44.550256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:44.553189 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 390/789] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:13:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:44.561060 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c956d5da-e068-4079-8dfa-da13aa3f8fb9 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:44.651222 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:44.665259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a3a6a7-0703-4369-b380-8e98600cb509 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:44.713080 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2255bfc5-dfe6-4907-bc92-0383761c2ea8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:44.713654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2255bfc5-dfe6-4907-bc92-0383761c2ea8 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:44.714141 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2255bfc5-dfe6-4907-bc92-0383761c2ea8 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:44.714615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2255bfc5-dfe6-4907-bc92-0383761c2ea8 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:44.729986 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2255bfc5-dfe6-4907-bc92-0383761c2ea8 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:44.731767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2255bfc5-dfe6-4907-bc92-0383761c2ea8 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:44.754123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-2255bfc5-dfe6-4907-bc92-0383761c2ea8 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 04:13:44.754123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:44.754123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:44.754123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:44.754123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:44.754123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:44.754123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:44.754123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:44.754123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:44.754123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:44.754123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:44.754123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:44.754123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:44.754123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:44.754123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:44.754123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:44.755148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:44.755148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:44.755148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:44.755148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:44.755148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:44.755148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:44.755148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:44.755148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:44.755148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:44.755148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:44.755148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:44.755148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:44.755148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:44.755148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:44.755148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:44.755148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:44.755148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:44.755148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:44.756334 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:44.756334 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:44.756334 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:44.756334 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:44.756334 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:44.756334 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:44.756334 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:44.756334 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:44.756334 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 04:13:44.756334 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:44.762325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 400/790] 10.222.0.85 () {56 vars in 1117 bytes} [Tue Aug 30 04:13:44 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:44.768276 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2255bfc5-dfe6-4907-bc92-0383761c2ea8 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:44.773377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:44.774930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c956d5da-e068-4079-8dfa-da13aa3f8fb9 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:44.921010 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2255bfc5-dfe6-4907-bc92-0383761c2ea8 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:44.956293 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6354f0f7-99bd-40f6-8228-f590215a1109 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:44.956923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6354f0f7-99bd-40f6-8228-f590215a1109 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:44.957225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6354f0f7-99bd-40f6-8228-f590215a1109 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:44.957993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6354f0f7-99bd-40f6-8228-f590215a1109 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:44.958737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6354f0f7-99bd-40f6-8228-f590215a1109 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:44.959040 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6354f0f7-99bd-40f6-8228-f590215a1109 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:44.965065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6354f0f7-99bd-40f6-8228-f590215a1109 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:44.981957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 401/791] 10.222.0.85 () {56 vars in 1149 bytes} [Tue Aug 30 04:13:44 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:44.987802 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6354f0f7-99bd-40f6-8228-f590215a1109 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:45.021944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e6c57d-3138-4995-a821-7d0b91ddb618 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:45.022714 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e6c57d-3138-4995-a821-7d0b91ddb618 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:45.023120 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e6c57d-3138-4995-a821-7d0b91ddb618 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:45.023518 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e6c57d-3138-4995-a821-7d0b91ddb618 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:45.024293 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e6c57d-3138-4995-a821-7d0b91ddb618 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:45.024696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e6c57d-3138-4995-a821-7d0b91ddb618 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:45.026409 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e6c57d-3138-4995-a821-7d0b91ddb618 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:45.036156 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 391/792] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:13:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 476 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:45.041723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e6c57d-3138-4995-a821-7d0b91ddb618 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:45.112998 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:45.158147 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6354f0f7-99bd-40f6-8228-f590215a1109 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:45.197377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea140f76-bfea-474a-8e46-51a1476fb004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:45.198440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea140f76-bfea-474a-8e46-51a1476fb004 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:45.198857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea140f76-bfea-474a-8e46-51a1476fb004 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:45.199140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea140f76-bfea-474a-8e46-51a1476fb004 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:45.211217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea140f76-bfea-474a-8e46-51a1476fb004 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:45.212861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea140f76-bfea-474a-8e46-51a1476fb004 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:45.230736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-ea140f76-bfea-474a-8e46-51a1476fb004 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:45.230736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:45.230736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:45.230736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:45.230736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:45.230736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:45.230736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:45.230736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:45.230736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:45.230736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:45.230736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:45.230736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:45.230736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:45.230736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:45.230736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:45.230736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:45.231887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:45.231887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:45.231887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:45.231887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:45.231887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:45.231887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:45.231887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:45.231887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:45.231887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:45.231887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:45.231887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:45.231887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:45.231887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:45.231887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:45.231887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:45.231887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:45.231887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:45.232781 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:45.232781 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:45.232781 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:45.232781 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 402/793] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:45.240971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea140f76-bfea-474a-8e46-51a1476fb004 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:45.319558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e6c57d-3138-4995-a821-7d0b91ddb618 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:45.351231 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:45.364232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea140f76-bfea-474a-8e46-51a1476fb004 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:45.385441 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:45.405761 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec01b6e-c3a3-4472-a0c5-b699e5295133 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:45.406752 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec01b6e-c3a3-4472-a0c5-b699e5295133 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:45.407237 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec01b6e-c3a3-4472-a0c5-b699e5295133 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:45.407735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec01b6e-c3a3-4472-a0c5-b699e5295133 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:45.408596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec01b6e-c3a3-4472-a0c5-b699e5295133 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:45.409129 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec01b6e-c3a3-4472-a0c5-b699e5295133 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:45.410997 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec01b6e-c3a3-4472-a0c5-b699e5295133 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:45.426084 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 403/794] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:45.432056 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec01b6e-c3a3-4472-a0c5-b699e5295133 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:45.469987 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb62d9b5-9c58-45b4-8d6b-de6311e5e5a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:45.470578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb62d9b5-9c58-45b4-8d6b-de6311e5e5a7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:45.470882 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb62d9b5-9c58-45b4-8d6b-de6311e5e5a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:45.471132 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb62d9b5-9c58-45b4-8d6b-de6311e5e5a7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:45.514079 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb62d9b5-9c58-45b4-8d6b-de6311e5e5a7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:45.515405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb62d9b5-9c58-45b4-8d6b-de6311e5e5a7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:45.532135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-eb62d9b5-9c58-45b4-8d6b-de6311e5e5a7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:45.532135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:45.532135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:45.532135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:45.532135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:45.532135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:45.532135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:45.532135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:45.532135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:45.532135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:45.532135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:45.532135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:45.532135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:45.532135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:45.532135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:45.532135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:45.533234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:45.533234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:45.533234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:45.533234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:45.533234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:45.533234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:45.533234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:45.533234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:45.533234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:45.533234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:45.533234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:45.533234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:45.533234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:45.533234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:45.533234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:45.533234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:45.533234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:45.534114 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:45.534114 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:45.534114 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:45.538739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 392/795] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 492 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:45.540695 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb62d9b5-9c58-45b4-8d6b-de6311e5e5a7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:45.555356 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec01b6e-c3a3-4472-a0c5-b699e5295133 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:45.635484 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92096788-d0c7-44a5-9195-36d4f47f7d73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:45.636423 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92096788-d0c7-44a5-9195-36d4f47f7d73 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:45.636856 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92096788-d0c7-44a5-9195-36d4f47f7d73 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:45.637322 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92096788-d0c7-44a5-9195-36d4f47f7d73 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:45.648563 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92096788-d0c7-44a5-9195-36d4f47f7d73 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:45.651033 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92096788-d0c7-44a5-9195-36d4f47f7d73 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:45.671520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-92096788-d0c7-44a5-9195-36d4f47f7d73 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:45.671520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:45.671520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:45.671520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:45.671520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:45.671520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:45.671520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:45.671520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:45.671520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:45.671520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:45.671520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:45.671520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:45.671520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:45.671520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:45.671520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:45.672475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:45.672475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:45.672475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:45.672475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:45.672475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:45.672475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:45.672475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:45.672475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:45.672475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:45.672475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:45.672475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:45.672475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:45.672475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:45.672475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:45.672475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:45.672475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:45.672475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:45.673396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:45.673396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:45.673396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:45.673396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:45.673396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:45.673396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:45.673396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:45.673396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:45.674313 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 404/796] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:45.675990 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d899e218-3e2a-43fd-82fc-a26a857e90ee None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:45.676521 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d899e218-3e2a-43fd-82fc-a26a857e90ee None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:45.676910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d899e218-3e2a-43fd-82fc-a26a857e90ee None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:45.679300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 405/797] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:45 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:45.680196 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d899e218-3e2a-43fd-82fc-a26a857e90ee None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:45.701216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb62d9b5-9c58-45b4-8d6b-de6311e5e5a7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:45.727997 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:45.753986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e28ec0-062f-45e2-b062-2797174010b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:45.755067 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e28ec0-062f-45e2-b062-2797174010b0 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:45.755556 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e28ec0-062f-45e2-b062-2797174010b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:45.756058 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e28ec0-062f-45e2-b062-2797174010b0 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:45.757686 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e28ec0-062f-45e2-b062-2797174010b0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:45.757995 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e28ec0-062f-45e2-b062-2797174010b0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:45.759390 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e28ec0-062f-45e2-b062-2797174010b0 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:45.781189 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 393/798] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:45.783686 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65879863-cc53-4fa9-b6f9-17932444fcc0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:45.783901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65879863-cc53-4fa9-b6f9-17932444fcc0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:45.784288 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65879863-cc53-4fa9-b6f9-17932444fcc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:45.805663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-65879863-cc53-4fa9-b6f9-17932444fcc0 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:45.807828 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-65879863-cc53-4fa9-b6f9-17932444fcc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:45.828140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d899e218-3e2a-43fd-82fc-a26a857e90ee None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:45.830328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:45.879319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663ded7b-6b83-4f2a-af15-a400eddac62c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:45.881774 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663ded7b-6b83-4f2a-af15-a400eddac62c None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:45.882047 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663ded7b-6b83-4f2a-af15-a400eddac62c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:45.882314 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663ded7b-6b83-4f2a-af15-a400eddac62c None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:45.883018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663ded7b-6b83-4f2a-af15-a400eddac62c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:45.883318 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663ded7b-6b83-4f2a-af15-a400eddac62c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:45.884755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663ded7b-6b83-4f2a-af15-a400eddac62c None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:45.898301 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 406/799] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:45.899274 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663ded7b-6b83-4f2a-af15-a400eddac62c None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:45.901635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 394/800] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:45 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:13:45.903734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65879863-cc53-4fa9-b6f9-17932444fcc0 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:46.000496 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:46.009018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663ded7b-6b83-4f2a-af15-a400eddac62c None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:46.019270 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65879863-cc53-4fa9-b6f9-17932444fcc0 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:46.047554 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9a26ff-6da1-4846-a38b-733627a0273e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:46.048182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9a26ff-6da1-4846-a38b-733627a0273e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:46.048475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9a26ff-6da1-4846-a38b-733627a0273e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:46.048870 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9a26ff-6da1-4846-a38b-733627a0273e None admin] PATH_INFO: `/v3/users/swift` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:46.058140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9a26ff-6da1-4846-a38b-733627a0273e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:46.058408 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9ecdca-19be-48d8-b3a0-adc632121930 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:46.059318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9ecdca-19be-48d8-b3a0-adc632121930 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:46.059612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9a26ff-6da1-4846-a38b-733627a0273e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:46.060007 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9ecdca-19be-48d8-b3a0-adc632121930 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:46.060483 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9ecdca-19be-48d8-b3a0-adc632121930 None admin] PATH_INFO: `/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/00aeab772672403984bdcb8baaa1520e/roles/755eb1bfc0cf445d8f282358f404a320` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:46.073639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-ad9a26ff-6da1-4846-a38b-733627a0273e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:13:46.073639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:46.073639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:46.073639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:46.073639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:46.073639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:46.073639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:46.073639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:46.073639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:46.073639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:46.073639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:46.073639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:46.073639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:46.073639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:46.073639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:46.073639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:46.074635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:46.074635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:46.074635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:46.074635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:46.074635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:46.074635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:46.074635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:46.074635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:46.074635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:46.074635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:46.074635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:46.074635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:46.074635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:46.074635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:46.074635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:46.074635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:46.074635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:46.074635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:46.075571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:46.075571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:46.075571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:46.075571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:46.075571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:46.075571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:46.075571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:46.075571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:46.075571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:13:46.075571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:46.077117 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 407/801] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:13:45 2022] GET /identity/v3/users/swift => generated 83 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:46.081126 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9a26ff-6da1-4846-a38b-733627a0273e None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:46.082660 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9ecdca-19be-48d8-b3a0-adc632121930 None admin] RBAC: Authorizing `identity:create_grant(project_id=a860dcc3b2d64d1491c40710ec7423a5, user_id=00aeab772672403984bdcb8baaa1520e, role_id=755eb1bfc0cf445d8f282358f404a320)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:46.084124 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9ecdca-19be-48d8-b3a0-adc632121930 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:46.101160 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 395/802] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:13:45 2022] PUT /identity/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/00aeab772672403984bdcb8baaa1520e/roles/755eb1bfc0cf445d8f282358f404a320 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:13:46.206648 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9a26ff-6da1-4846-a38b-733627a0273e None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:46.244785 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729da387-7973-4ce0-bc93-e2b7eb6510b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:46.245798 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729da387-7973-4ce0-bc93-e2b7eb6510b1 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:46.246254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729da387-7973-4ce0-bc93-e2b7eb6510b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:46.246699 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729da387-7973-4ce0-bc93-e2b7eb6510b1 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:46.247613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729da387-7973-4ce0-bc93-e2b7eb6510b1 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:46.248090 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729da387-7973-4ce0-bc93-e2b7eb6510b1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:46.249753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729da387-7973-4ce0-bc93-e2b7eb6510b1 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:46.271246 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 408/803] 10.222.0.85 () {56 vars in 1135 bytes} [Tue Aug 30 04:13:46 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:46.276735 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9ecdca-19be-48d8-b3a0-adc632121930 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:46.281818 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:46.335279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729da387-7973-4ce0-bc93-e2b7eb6510b1 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:46.354506 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:46.407895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9ecdca-19be-48d8-b3a0-adc632121930 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:46.443990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c788d604-3593-40e2-8f42-07e975b293d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:46.444725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c788d604-3593-40e2-8f42-07e975b293d5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:46.445125 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c788d604-3593-40e2-8f42-07e975b293d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:46.445533 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c788d604-3593-40e2-8f42-07e975b293d5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:46.453701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729da387-7973-4ce0-bc93-e2b7eb6510b1 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:46.456421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c788d604-3593-40e2-8f42-07e975b293d5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:46.458450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c788d604-3593-40e2-8f42-07e975b293d5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:46.473563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-c788d604-3593-40e2-8f42-07e975b293d5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:46.473563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:46.473563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:46.473563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:46.473563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:46.473563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:46.473563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:46.473563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:46.473563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:46.473563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:46.473563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:46.473563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:46.473563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:46.473563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:46.473563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:46.473563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:46.474516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:46.474516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:46.474516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:46.474516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:46.474516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:46.474516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:46.474516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:46.474516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:46.474516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:46.474516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:46.474516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:46.474516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:46.474516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:46.474516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:46.474516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:46.474516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:46.474516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:46.475409 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:46.475409 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:46.475409 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:46.479391 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 396/804] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:46.480670 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138e6c9c-2c46-47f2-9de0-07b73448f210 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:46.480951 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138e6c9c-2c46-47f2-9de0-07b73448f210 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:46.481241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138e6c9c-2c46-47f2-9de0-07b73448f210 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:46.486076 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 397/805] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:46.486076 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138e6c9c-2c46-47f2-9de0-07b73448f210 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:46.500229 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4767cfdf-e766-4319-bf2a-c6b318ba5563 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:46.501382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4767cfdf-e766-4319-bf2a-c6b318ba5563 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:46.501906 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4767cfdf-e766-4319-bf2a-c6b318ba5563 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:46.502394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4767cfdf-e766-4319-bf2a-c6b318ba5563 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:46.503171 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4767cfdf-e766-4319-bf2a-c6b318ba5563 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:46.504973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4767cfdf-e766-4319-bf2a-c6b318ba5563 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:46.515225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-4767cfdf-e766-4319-bf2a-c6b318ba5563 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 04:13:46.515225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:46.515225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:13:46.515225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:13:46.515225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:13:46.515225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:13:46.515225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:13:46.515225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:46.515225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 04:13:46.515225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:46.515225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:13:46.515225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:46.515225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:46.515225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:46.515225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:46.515225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:46.515225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:46.516231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:46.516231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:46.516231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:46.516231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:46.516231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:46.516231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:46.516231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:13:46.516231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:13:46.516231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:13:46.516231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:13:46.516231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:46.516231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:46.516231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:46.516231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:46.516231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:46.516231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:46.517140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:46.517140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:46.517140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:46.517140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:46.517140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:46.517140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:46.517140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:46.517140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:46.517140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:46.517140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:46.517140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:13:46.517140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:46.517140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 04:13:46.517140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:46.520711 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 409/806] 10.222.0.85 () {56 vars in 1129 bytes} [Tue Aug 30 04:13:46 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:46.522324 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bb42a5-d9c6-46d0-849e-542b7462d739 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:46.522863 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bb42a5-d9c6-46d0-849e-542b7462d739 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:46.523479 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bb42a5-d9c6-46d0-849e-542b7462d739 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:46.545168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:46.548509 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-76bb42a5-d9c6-46d0-849e-542b7462d739 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:46.549746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-76bb42a5-d9c6-46d0-849e-542b7462d739 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:46.607957 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138e6c9c-2c46-47f2-9de0-07b73448f210 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:46.633621 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 410/807] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:46 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:13:46.633621 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bb42a5-d9c6-46d0-849e-542b7462d739 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:46.644845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:46.673921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b362f4c-8951-4cfa-a26e-81e37b87d31f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:46.674685 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b362f4c-8951-4cfa-a26e-81e37b87d31f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:46.676491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b362f4c-8951-4cfa-a26e-81e37b87d31f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:46.676928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b362f4c-8951-4cfa-a26e-81e37b87d31f None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:46.677982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b362f4c-8951-4cfa-a26e-81e37b87d31f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:46.678311 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b362f4c-8951-4cfa-a26e-81e37b87d31f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:46.680273 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b362f4c-8951-4cfa-a26e-81e37b87d31f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:46.706528 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 398/808] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:46.715893 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b362f4c-8951-4cfa-a26e-81e37b87d31f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:46.797908 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bb42a5-d9c6-46d0-849e-542b7462d739 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:46.839134 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1323172d-7d29-4eb4-9c19-7d9e2c212872 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:46.839536 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1323172d-7d29-4eb4-9c19-7d9e2c212872 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:46.839840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1323172d-7d29-4eb4-9c19-7d9e2c212872 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:46.840094 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1323172d-7d29-4eb4-9c19-7d9e2c212872 None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:46.840779 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1323172d-7d29-4eb4-9c19-7d9e2c212872 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:46.841104 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1323172d-7d29-4eb4-9c19-7d9e2c212872 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:46.842478 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1323172d-7d29-4eb4-9c19-7d9e2c212872 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:46.852842 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b362f4c-8951-4cfa-a26e-81e37b87d31f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:46.853371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 411/809] 10.222.0.85 () {56 vars in 1124 bytes} [Tue Aug 30 04:13:46 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:46.854723 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9887b51b-811d-46cb-b25b-99c9065659ad None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:46.855004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9887b51b-811d-46cb-b25b-99c9065659ad None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:46.855276 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9887b51b-811d-46cb-b25b-99c9065659ad None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:46.856799 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 412/810] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:46.857532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9887b51b-811d-46cb-b25b-99c9065659ad None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:46.869859 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:46.893213 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d72026d-a759-4124-9e68-08c58febc5ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:46.894214 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d72026d-a759-4124-9e68-08c58febc5ad None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:46.894635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d72026d-a759-4124-9e68-08c58febc5ad None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:46.895079 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d72026d-a759-4124-9e68-08c58febc5ad None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:46.902714 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d72026d-a759-4124-9e68-08c58febc5ad None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:46.904512 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d72026d-a759-4124-9e68-08c58febc5ad None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:46.917207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-6d72026d-a759-4124-9e68-08c58febc5ad None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:46.917207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:46.917207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:46.917207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:46.917207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:46.917207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:46.917207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:46.917207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:46.917207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:46.917207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:46.917207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:46.917207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:46.917207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:46.917207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:46.917207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:46.918261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:46.918261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:46.918261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:46.918261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:46.918261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:46.918261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:46.918261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:46.918261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:46.918261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:46.918261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:46.918261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:46.918261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:46.918261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:46.918261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:46.918261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:46.918261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:46.918261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:46.919244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:46.919244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:46.919244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:46.919244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:46.919244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:46.919244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:46.919244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:46.919244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:46.921371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 399/811] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:46.922855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e26cb3c-d5c0-4a39-a2ae-40d5e245a2a1 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:46.923316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e26cb3c-d5c0-4a39-a2ae-40d5e245a2a1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:46.923800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e26cb3c-d5c0-4a39-a2ae-40d5e245a2a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:46.935998 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:46.945877 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5e26cb3c-d5c0-4a39-a2ae-40d5e245a2a1 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:46.947444 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5e26cb3c-d5c0-4a39-a2ae-40d5e245a2a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:46.969490 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9887b51b-811d-46cb-b25b-99c9065659ad None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:47.005721 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7cd4d1c-59e8-417f-a5ed-58f1ad0a9075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:47.006588 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cd4d1c-59e8-417f-a5ed-58f1ad0a9075 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:47.006835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cd4d1c-59e8-417f-a5ed-58f1ad0a9075 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:47.007102 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cd4d1c-59e8-417f-a5ed-58f1ad0a9075 None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:47.007977 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7cd4d1c-59e8-417f-a5ed-58f1ad0a9075 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:47.008112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7cd4d1c-59e8-417f-a5ed-58f1ad0a9075 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:47.009491 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7cd4d1c-59e8-417f-a5ed-58f1ad0a9075 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:47.019083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 413/812] 10.222.0.85 () {56 vars in 1124 bytes} [Tue Aug 30 04:13:46 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:47.019083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7cd4d1c-59e8-417f-a5ed-58f1ad0a9075 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:47.023408 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 400/813] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:46 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:13:47.024208 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e26cb3c-d5c0-4a39-a2ae-40d5e245a2a1 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:47.142765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7cd4d1c-59e8-417f-a5ed-58f1ad0a9075 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:47.156401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e26cb3c-d5c0-4a39-a2ae-40d5e245a2a1 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:47.160097 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:47.181220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5f3cf3-bf61-4242-8919-4509b2668118 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:47.181823 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5f3cf3-bf61-4242-8919-4509b2668118 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:47.182099 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5f3cf3-bf61-4242-8919-4509b2668118 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:47.182596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5f3cf3-bf61-4242-8919-4509b2668118 None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:47.183300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5f3cf3-bf61-4242-8919-4509b2668118 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:47.183635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5f3cf3-bf61-4242-8919-4509b2668118 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:47.185212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5f3cf3-bf61-4242-8919-4509b2668118 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:47.186905 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:47.204658 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313f85f6-d0f2-49de-a52b-67b56036108e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:47.205005 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 414/814] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:47.205778 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5f3cf3-bf61-4242-8919-4509b2668118 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:47.206258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313f85f6-d0f2-49de-a52b-67b56036108e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:47.206729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313f85f6-d0f2-49de-a52b-67b56036108e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:47.207202 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313f85f6-d0f2-49de-a52b-67b56036108e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:47.223022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313f85f6-d0f2-49de-a52b-67b56036108e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:47.230271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-313f85f6-d0f2-49de-a52b-67b56036108e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:13:47.230271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:47.230271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:47.230271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:47.230271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:47.230271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:47.230271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:47.230271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:47.230271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:47.230271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:47.230271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:47.230271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:47.230271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:47.230271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:47.230271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:47.230271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:47.231186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:47.231186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:47.231186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:47.231186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:47.231186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:47.231186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:47.231186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:47.231186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:47.231186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:47.231186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:47.231186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:47.231186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:47.231186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:47.231186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:47.231186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:47.231186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:47.232028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:47.232028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:47.232028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:47.232028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:47.232028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:47.232028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:47.232028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:47.232028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:47.232028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:47.232028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:47.232028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:13:47.232028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:47.234211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 401/815] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:13:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:47.235092 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313f85f6-d0f2-49de-a52b-67b56036108e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:47.343101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313f85f6-d0f2-49de-a52b-67b56036108e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:47.343434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5f3cf3-bf61-4242-8919-4509b2668118 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:47.383067 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a13416e-34a1-44ab-8c79-831f28c42ed6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:47.383295 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662e5662-d10c-41a3-a862-dc7d5b2fe3d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:47.383653 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a13416e-34a1-44ab-8c79-831f28c42ed6 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:47.383930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a13416e-34a1-44ab-8c79-831f28c42ed6 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:47.384201 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a13416e-34a1-44ab-8c79-831f28c42ed6 None admin] PATH_INFO: `/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/6cb539ef6a69406baa70f660e1a815b4/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:47.384505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662e5662-d10c-41a3-a862-dc7d5b2fe3d7 None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:47.384939 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662e5662-d10c-41a3-a862-dc7d5b2fe3d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:47.385235 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662e5662-d10c-41a3-a862-dc7d5b2fe3d7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:47.385899 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662e5662-d10c-41a3-a862-dc7d5b2fe3d7 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:47.386286 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a13416e-34a1-44ab-8c79-831f28c42ed6 None admin] RBAC: Authorizing `identity:create_grant(project_id=a860dcc3b2d64d1491c40710ec7423a5, user_id=6cb539ef6a69406baa70f660e1a815b4, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:47.387264 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662e5662-d10c-41a3-a862-dc7d5b2fe3d7 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:47.387711 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a13416e-34a1-44ab-8c79-831f28c42ed6 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:47.402301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 402/816] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:13:47 2022] PUT /identity/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/6cb539ef6a69406baa70f660e1a815b4/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:13:47.403135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a13416e-34a1-44ab-8c79-831f28c42ed6 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:47.426156 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 415/817] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:13:47 2022] POST /identity/v3/endpoints => generated 346 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:13:47.428146 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662e5662-d10c-41a3-a862-dc7d5b2fe3d7 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:47.450839 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:47.500504 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:47.582640 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662e5662-d10c-41a3-a862-dc7d5b2fe3d7 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:47.591661 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a13416e-34a1-44ab-8c79-831f28c42ed6 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:47.639149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ceb6cc7-5e05-4c0c-aa6d-85d2e26d8746 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:47.640455 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ceb6cc7-5e05-4c0c-aa6d-85d2e26d8746 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:47.641030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ceb6cc7-5e05-4c0c-aa6d-85d2e26d8746 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:47.641448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cca6cdf-a25d-4324-bc03-7015fff71de0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:47.641785 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ceb6cc7-5e05-4c0c-aa6d-85d2e26d8746 None admin] PATH_INFO: `/v3/services/network` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:47.642130 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cca6cdf-a25d-4324-bc03-7015fff71de0 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:47.642411 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cca6cdf-a25d-4324-bc03-7015fff71de0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:47.642676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cca6cdf-a25d-4324-bc03-7015fff71de0 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:47.643447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ceb6cc7-5e05-4c0c-aa6d-85d2e26d8746 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:47.643644 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cca6cdf-a25d-4324-bc03-7015fff71de0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:47.644106 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cca6cdf-a25d-4324-bc03-7015fff71de0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:47.645692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cca6cdf-a25d-4324-bc03-7015fff71de0 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:47.645866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ceb6cc7-5e05-4c0c-aa6d-85d2e26d8746 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:47.655548 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 403/818] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:13:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:47.658163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-4ceb6cc7-5e05-4c0c-aa6d-85d2e26d8746 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 04:13:47.658163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:47.658163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:13:47.658163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:13:47.658163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:13:47.658163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:13:47.658163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:13:47.658163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:47.658163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 04:13:47.658163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:47.658163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:13:47.658163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:47.658163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:47.658163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:47.658163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:47.658163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:47.658163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:47.659312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:47.659312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:47.659312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:47.659312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:47.659312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:47.659312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:47.659312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:13:47.659312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:13:47.659312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:13:47.659312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:13:47.659312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:47.659312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:47.659312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:47.659312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:47.659312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:47.659312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:47.661613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:47.661613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:47.661613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:47.661613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:47.661613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:47.661613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:47.661613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:47.661613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:47.661613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:47.661613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:47.661613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:13:47.661613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:47.661613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 04:13:47.661613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:47.661613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 416/819] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:47 2022] GET /identity/v3/services/network => generated 88 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:47.663680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cca6cdf-a25d-4324-bc03-7015fff71de0 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:47.666798 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ceb6cc7-5e05-4c0c-aa6d-85d2e26d8746 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:47.773535 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:47.782792 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cca6cdf-a25d-4324-bc03-7015fff71de0 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:47.804197 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ceb6cc7-5e05-4c0c-aa6d-85d2e26d8746 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:47.820148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:47.836168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0423c5ee-73b7-4bc3-a77a-1ce9eb505e68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:47.836828 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0423c5ee-73b7-4bc3-a77a-1ce9eb505e68 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:47.837151 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0423c5ee-73b7-4bc3-a77a-1ce9eb505e68 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:47.837427 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0423c5ee-73b7-4bc3-a77a-1ce9eb505e68 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:47.853415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0423c5ee-73b7-4bc3-a77a-1ce9eb505e68 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:47.854210 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cd5c0f-c0ab-4271-aa91-49015a5931ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:47.854885 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0423c5ee-73b7-4bc3-a77a-1ce9eb505e68 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:47.855265 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cd5c0f-c0ab-4271-aa91-49015a5931ce None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:47.855733 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cd5c0f-c0ab-4271-aa91-49015a5931ce None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:47.856262 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cd5c0f-c0ab-4271-aa91-49015a5931ce None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:47.857323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cd5c0f-c0ab-4271-aa91-49015a5931ce None admin] RBAC: Adding query filter params (name=network) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:47.857847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cd5c0f-c0ab-4271-aa91-49015a5931ce None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:47.859385 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cd5c0f-c0ab-4271-aa91-49015a5931ce None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:47.865143 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 417/820] 10.222.0.85 () {56 vars in 1112 bytes} [Tue Aug 30 04:13:47 2022] GET /identity/v3/services?name=network => generated 124 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:47.871814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cd5c0f-c0ab-4271-aa91-49015a5931ce None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:47.877671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-0423c5ee-73b7-4bc3-a77a-1ce9eb505e68 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:47.877671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:47.877671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:47.877671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:47.877671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:47.877671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:47.877671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:47.877671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:47.877671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:47.877671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:47.877671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:47.877671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:47.877671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:47.877671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:47.877671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:47.877671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:47.878598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:47.878598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:47.878598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:47.878598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:47.878598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:47.878598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:47.878598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:47.878598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:47.878598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:47.878598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:47.878598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:47.878598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:47.878598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:47.878598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:47.878598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:47.878598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:47.878598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:47.880625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:47.880625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:47.880625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:47.880625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 404/821] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:47.886260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0423c5ee-73b7-4bc3-a77a-1ce9eb505e68 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:48.018496 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0423c5ee-73b7-4bc3-a77a-1ce9eb505e68 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:48.059949 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f46404-4e4b-4f57-a3a5-5f630d40037f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:48.060829 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f46404-4e4b-4f57-a3a5-5f630d40037f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:48.061146 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f46404-4e4b-4f57-a3a5-5f630d40037f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:48.061427 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f46404-4e4b-4f57-a3a5-5f630d40037f None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:48.062078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f46404-4e4b-4f57-a3a5-5f630d40037f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:48.062385 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f46404-4e4b-4f57-a3a5-5f630d40037f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:48.064006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f46404-4e4b-4f57-a3a5-5f630d40037f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:48.072423 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:48.078881 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cd5c0f-c0ab-4271-aa91-49015a5931ce None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:48.079951 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 405/822] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:48.090258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f46404-4e4b-4f57-a3a5-5f630d40037f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:48.136128 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73456b8-5b83-4ba4-81c7-be7cd3cba809 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:48.141749 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73456b8-5b83-4ba4-81c7-be7cd3cba809 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:48.142256 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73456b8-5b83-4ba4-81c7-be7cd3cba809 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:48.142738 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73456b8-5b83-4ba4-81c7-be7cd3cba809 None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:48.143618 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73456b8-5b83-4ba4-81c7-be7cd3cba809 None admin] RBAC: Adding query filter params (type=network) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:48.144131 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73456b8-5b83-4ba4-81c7-be7cd3cba809 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:48.145899 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73456b8-5b83-4ba4-81c7-be7cd3cba809 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:48.156367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 418/823] 10.222.0.85 () {56 vars in 1112 bytes} [Tue Aug 30 04:13:47 2022] GET /identity/v3/services?type=network => generated 350 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:48.162109 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73456b8-5b83-4ba4-81c7-be7cd3cba809 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:48.223648 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f46404-4e4b-4f57-a3a5-5f630d40037f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:48.264363 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92dbc70-77c1-472e-8054-db13cb13707f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:48.265025 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92dbc70-77c1-472e-8054-db13cb13707f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:48.265321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92dbc70-77c1-472e-8054-db13cb13707f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:48.265590 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92dbc70-77c1-472e-8054-db13cb13707f None admin] PATH_INFO: `/v3/users/nova` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:48.267237 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:48.274913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92dbc70-77c1-472e-8054-db13cb13707f None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:48.276365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92dbc70-77c1-472e-8054-db13cb13707f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:48.289431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-f92dbc70-77c1-472e-8054-db13cb13707f None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:13:48.289431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:48.289431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:48.289431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:48.289431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:48.289431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:48.289431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:48.289431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:48.289431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:48.289431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:48.289431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:48.289431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:48.289431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:48.289431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:48.289431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:48.289431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:48.290393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:48.290393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:48.290393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:48.290393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:48.290393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:48.290393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:48.290393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:48.290393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:48.290393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:48.290393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:48.290393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:48.290393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:48.290393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:48.290393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:48.290393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:48.290393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:48.290393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:48.290393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:48.291352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:48.291352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:48.291352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:48.291352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:48.291352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:48.291352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:48.291352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:48.291352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:48.291352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:13:48.291352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:48.291352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 406/824] 10.222.0.85 () {56 vars in 1093 bytes} [Tue Aug 30 04:13:48 2022] GET /identity/v3/users/nova => generated 82 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:48.292429 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2ced92-3a39-4d39-aa2f-bbe85f5dd851 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:48.296558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2ced92-3a39-4d39-aa2f-bbe85f5dd851 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:48.296558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2ced92-3a39-4d39-aa2f-bbe85f5dd851 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:48.296558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 407/825] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:48.296558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2ced92-3a39-4d39-aa2f-bbe85f5dd851 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:48.342919 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73456b8-5b83-4ba4-81c7-be7cd3cba809 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:48.378294 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97b0b52-8b1a-4130-ac2d-78b185b46c4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:48.379019 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97b0b52-8b1a-4130-ac2d-78b185b46c4e None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:48.379879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97b0b52-8b1a-4130-ac2d-78b185b46c4e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:48.380228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97b0b52-8b1a-4130-ac2d-78b185b46c4e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:48.380916 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:48.381323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97b0b52-8b1a-4130-ac2d-78b185b46c4e None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:48.382741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97b0b52-8b1a-4130-ac2d-78b185b46c4e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:48.413815 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 419/826] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:13:48 2022] POST /identity/v3/endpoints => generated 337 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:13:48.418309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1448b0e0-6c39-40d7-8579-bf94a261f791 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:48.418915 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1448b0e0-6c39-40d7-8579-bf94a261f791 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:48.419335 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1448b0e0-6c39-40d7-8579-bf94a261f791 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:48.458790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-1448b0e0-6c39-40d7-8579-bf94a261f791 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:48.460032 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-1448b0e0-6c39-40d7-8579-bf94a261f791 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:48.463718 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2ced92-3a39-4d39-aa2f-bbe85f5dd851 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:48.503630 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c86b7b8-5378-48b1-bbc5-7fe55e40cfd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:48.505219 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c86b7b8-5378-48b1-bbc5-7fe55e40cfd7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:48.505784 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c86b7b8-5378-48b1-bbc5-7fe55e40cfd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:48.506542 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c86b7b8-5378-48b1-bbc5-7fe55e40cfd7 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:48.507516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c86b7b8-5378-48b1-bbc5-7fe55e40cfd7 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:48.508037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c86b7b8-5378-48b1-bbc5-7fe55e40cfd7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:48.509769 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c86b7b8-5378-48b1-bbc5-7fe55e40cfd7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:48.531490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 408/827] 10.222.0.85 () {56 vars in 1133 bytes} [Tue Aug 30 04:13:48 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:48.537594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c86b7b8-5378-48b1-bbc5-7fe55e40cfd7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:48.638869 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 420/828] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:48 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:13:48.664145 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c86b7b8-5378-48b1-bbc5-7fe55e40cfd7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:48.678896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:48.702113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0a7652-a565-4994-ac09-34213540be85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:48.702732 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0a7652-a565-4994-ac09-34213540be85 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:48.703005 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0a7652-a565-4994-ac09-34213540be85 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:48.703274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0a7652-a565-4994-ac09-34213540be85 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:48.783444 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0a7652-a565-4994-ac09-34213540be85 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:48.784839 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0a7652-a565-4994-ac09-34213540be85 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:48.888233 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 409/829] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:48 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 04:13:48.893471 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1448b0e0-6c39-40d7-8579-bf94a261f791 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:48.922581 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:48.952217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0a7652-a565-4994-ac09-34213540be85 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:48.967603 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:49.017945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1448b0e0-6c39-40d7-8579-bf94a261f791 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:49.061102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0a7652-a565-4994-ac09-34213540be85 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:49.066194 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1491f325-2155-4f2d-a574-693378994972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:49.067339 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1491f325-2155-4f2d-a574-693378994972 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:49.067633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1491f325-2155-4f2d-a574-693378994972 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:49.067900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1491f325-2155-4f2d-a574-693378994972 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:49.081482 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1491f325-2155-4f2d-a574-693378994972 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:49.083543 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1491f325-2155-4f2d-a574-693378994972 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:49.110397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-1491f325-2155-4f2d-a574-693378994972 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:49.110397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:49.110397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:49.110397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:49.110397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:49.110397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:49.110397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:49.110397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:49.110397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:49.110397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:49.110397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:49.110397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:49.110397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:49.110397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:49.110397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:49.110397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:49.111637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:49.111637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:49.111637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:49.111637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:49.111637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:49.111637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:49.111637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:49.111637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:49.111637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:49.111637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:49.111637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:49.111637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:49.111637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:49.111637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:49.111637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:49.111637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:49.111637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:49.112631 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:49.112631 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:49.112631 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:49.112800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e27597a-2bfe-43fe-a43c-ed0e3ee26c41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:49.112800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e27597a-2bfe-43fe-a43c-ed0e3ee26c41 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:49.112800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e27597a-2bfe-43fe-a43c-ed0e3ee26c41 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:49.112800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e27597a-2bfe-43fe-a43c-ed0e3ee26c41 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:49.113100 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 421/830] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:49.119765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1491f325-2155-4f2d-a574-693378994972 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:49.136191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e27597a-2bfe-43fe-a43c-ed0e3ee26c41 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:49.142251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-6e27597a-2bfe-43fe-a43c-ed0e3ee26c41 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:49.142251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:49.142251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:49.142251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:49.142251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:49.142251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:49.142251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:49.142251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:49.142251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:49.142251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:49.142251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:49.142251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:49.142251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:49.142251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:49.142251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:49.142251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:49.144017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:49.144017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:49.144017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:49.144017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:49.144017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:49.144017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:49.144017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:49.144017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:49.144017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:49.144017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:49.144017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:49.144017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:49.144017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:49.144017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:49.144017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:49.144017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:49.145655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:49.145655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:49.145655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:49.145655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:49.145655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:49.145655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:49.145655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:49.145655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:49.145655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:49.145655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:49.145655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:13:49.145655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:49.145655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 410/831] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:13:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:49.150545 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e27597a-2bfe-43fe-a43c-ed0e3ee26c41 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:49.227226 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:49.254124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1491f325-2155-4f2d-a574-693378994972 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:49.256330 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:49.293559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e27597a-2bfe-43fe-a43c-ed0e3ee26c41 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:49.308901 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd60ef74-856e-487d-86d2-3fa294f1a8c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:49.309924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd60ef74-856e-487d-86d2-3fa294f1a8c4 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:49.310415 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd60ef74-856e-487d-86d2-3fa294f1a8c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:49.310975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd60ef74-856e-487d-86d2-3fa294f1a8c4 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:49.311982 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd60ef74-856e-487d-86d2-3fa294f1a8c4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:49.312477 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd60ef74-856e-487d-86d2-3fa294f1a8c4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:49.314064 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd60ef74-856e-487d-86d2-3fa294f1a8c4 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:49.331149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 422/832] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:49.336730 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd60ef74-856e-487d-86d2-3fa294f1a8c4 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:49.344573 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903ceedc-3b16-4c02-8351-a905a5c8c8dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:49.345542 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903ceedc-3b16-4c02-8351-a905a5c8c8dd None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:49.346020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903ceedc-3b16-4c02-8351-a905a5c8c8dd None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:49.346614 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903ceedc-3b16-4c02-8351-a905a5c8c8dd None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:49.347450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903ceedc-3b16-4c02-8351-a905a5c8c8dd None admin] RBAC: Adding query filter params (name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:49.347754 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903ceedc-3b16-4c02-8351-a905a5c8c8dd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:49.349685 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903ceedc-3b16-4c02-8351-a905a5c8c8dd None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:49.359261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 411/833] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:13:49 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:49.370770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903ceedc-3b16-4c02-8351-a905a5c8c8dd None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:49.467707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd60ef74-856e-487d-86d2-3fa294f1a8c4 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:49.482996 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903ceedc-3b16-4c02-8351-a905a5c8c8dd None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:49.505310 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e93526f-dbda-4bb2-b72a-e3ff97d2ad1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:49.506657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e93526f-dbda-4bb2-b72a-e3ff97d2ad1c None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:49.507266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e93526f-dbda-4bb2-b72a-e3ff97d2ad1c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:49.507741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e93526f-dbda-4bb2-b72a-e3ff97d2ad1c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:49.510966 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:49.517362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e93526f-dbda-4bb2-b72a-e3ff97d2ad1c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:49.518202 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742865f3-5773-4500-b9f5-8a10f1b2d35f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:49.519028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742865f3-5773-4500-b9f5-8a10f1b2d35f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:49.519311 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742865f3-5773-4500-b9f5-8a10f1b2d35f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:49.519580 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742865f3-5773-4500-b9f5-8a10f1b2d35f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:49.520553 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e93526f-dbda-4bb2-b72a-e3ff97d2ad1c None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:49.522153 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:49.532455 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742865f3-5773-4500-b9f5-8a10f1b2d35f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:49.533943 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742865f3-5773-4500-b9f5-8a10f1b2d35f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:49.536008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-7e93526f-dbda-4bb2-b72a-e3ff97d2ad1c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:49.536008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:49.536008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:49.536008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:49.536008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:49.536008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:49.536008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:49.536008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:49.536008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:49.536008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:49.536008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:49.536008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:49.536008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:49.536008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:49.536008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:49.536900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:49.536900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:49.536900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:49.536900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:49.536900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:49.536900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:49.536900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:49.536900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:49.536900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:49.536900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:49.536900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:49.536900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:49.536900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:49.536900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:49.536900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:49.536900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:49.536900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:49.537835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:49.537835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:49.537835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:49.537835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:49.537835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:49.537835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:49.537835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:49.537835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:49.540206 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 423/834] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:49.546102 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e93526f-dbda-4bb2-b72a-e3ff97d2ad1c None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:49.553959 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-742865f3-5773-4500-b9f5-8a10f1b2d35f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:49.553959 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:49.553959 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:49.553959 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:49.553959 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:49.553959 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:49.553959 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:49.553959 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:49.553959 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:49.553959 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:49.553959 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:49.553959 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:49.553959 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:49.553959 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:49.553959 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:49.553959 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:49.554913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:49.554913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:49.554913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:49.554913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:49.554913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:49.554913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:49.554913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:49.554913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:49.554913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:49.554913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:49.554913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:49.554913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:49.554913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:49.554913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:49.554913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:49.554913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:49.554913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:49.555800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:49.555800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:49.555800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:49.560042 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 412/835] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:49.560889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742865f3-5773-4500-b9f5-8a10f1b2d35f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:49.666526 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e93526f-dbda-4bb2-b72a-e3ff97d2ad1c None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:49.682412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742865f3-5773-4500-b9f5-8a10f1b2d35f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:49.703256 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63048190-1790-4b01-839b-c0903371422f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:49.703858 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63048190-1790-4b01-839b-c0903371422f None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:49.704133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63048190-1790-4b01-839b-c0903371422f None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:49.704400 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63048190-1790-4b01-839b-c0903371422f None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:49.705108 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63048190-1790-4b01-839b-c0903371422f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:49.705411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63048190-1790-4b01-839b-c0903371422f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:49.706838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63048190-1790-4b01-839b-c0903371422f None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:49.718175 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0d364f-81c5-4d55-8842-dc4ce9f6ca95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:49.718880 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0d364f-81c5-4d55-8842-dc4ce9f6ca95 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:49.725124 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0d364f-81c5-4d55-8842-dc4ce9f6ca95 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:49.725300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 424/836] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:49.725385 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0d364f-81c5-4d55-8842-dc4ce9f6ca95 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:49.725385 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0d364f-81c5-4d55-8842-dc4ce9f6ca95 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:49.725385 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0d364f-81c5-4d55-8842-dc4ce9f6ca95 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:49.725585 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8df12c-8dca-47a8-a9da-02df26f8482c None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:49.725585 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8df12c-8dca-47a8-a9da-02df26f8482c None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:49.725585 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8df12c-8dca-47a8-a9da-02df26f8482c None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:49.725771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0d364f-81c5-4d55-8842-dc4ce9f6ca95 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:49.725838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 425/837] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:49.725838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8df12c-8dca-47a8-a9da-02df26f8482c None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:49.739976 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 413/838] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:49.741551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47e4c33-ce35-420b-b419-29f103dd7e5a None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:49.742040 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47e4c33-ce35-420b-b419-29f103dd7e5a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:49.742510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47e4c33-ce35-420b-b419-29f103dd7e5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:49.761855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-e47e4c33-ce35-420b-b419-29f103dd7e5a None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:49.763042 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e47e4c33-ce35-420b-b419-29f103dd7e5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:49.792532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:49.809613 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:49.836457 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 414/839] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:49 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:13:49.845215 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47e4c33-ce35-420b-b419-29f103dd7e5a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:49.849021 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8df12c-8dca-47a8-a9da-02df26f8482c None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:49.887838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05a4e84-b93e-4659-aa38-94209e392681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:49.888560 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05a4e84-b93e-4659-aa38-94209e392681 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:49.888834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05a4e84-b93e-4659-aa38-94209e392681 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:49.889125 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05a4e84-b93e-4659-aa38-94209e392681 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:49.889843 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05a4e84-b93e-4659-aa38-94209e392681 None admin] RBAC: Adding query filter params (role.id=d17954e7ce544f0d90a512340232c60e, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=4bc4d455ebf041d488b1199ca58358a4) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:49.890142 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05a4e84-b93e-4659-aa38-94209e392681 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:49.891556 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05a4e84-b93e-4659-aa38-94209e392681 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:49.898039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 426/840] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:13:49 2022] GET /identity/v3/role_assignments?user.id=4bc4d455ebf041d488b1199ca58358a4&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=d17954e7ce544f0d90a512340232c60e => generated 608 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:49.918741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e87948-896e-45a0-84ad-b4f1b728ae02 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:49.918934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e87948-896e-45a0-84ad-b4f1b728ae02 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:49.919209 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e87948-896e-45a0-84ad-b4f1b728ae02 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:49.922087 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 427/841] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:49.926729 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8257fc4-34b3-4b7a-bc60-d94e53c25c16 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:49.927012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8257fc4-34b3-4b7a-bc60-d94e53c25c16 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:49.927272 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8257fc4-34b3-4b7a-bc60-d94e53c25c16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:49.947763 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-e8257fc4-34b3-4b7a-bc60-d94e53c25c16 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:49.949001 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e8257fc4-34b3-4b7a-bc60-d94e53c25c16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:49.957118 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47e4c33-ce35-420b-b419-29f103dd7e5a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:49.997112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64491c07-3535-4f67-af9a-0fb30686df79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:49.997769 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64491c07-3535-4f67-af9a-0fb30686df79 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:49.998045 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64491c07-3535-4f67-af9a-0fb30686df79 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:49.998436 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64491c07-3535-4f67-af9a-0fb30686df79 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:50.007709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64491c07-3535-4f67-af9a-0fb30686df79 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:50.009453 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64491c07-3535-4f67-af9a-0fb30686df79 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:50.024692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-64491c07-3535-4f67-af9a-0fb30686df79 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 04:13:50.024692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:50.024692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:50.024692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:50.024692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:50.024692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:50.024692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:50.024692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:50.024692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:50.024692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:50.024692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:50.024692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:50.024692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:50.024692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:50.024692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:50.024692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:50.025758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:50.025758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:50.025758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:50.025758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:50.025758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:50.025758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:50.025758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:50.025758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:50.025758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:50.025758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:50.025758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:50.025758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:50.025758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:50.025758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:50.025758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:50.025758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:50.025758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:50.025758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:50.027050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:50.027050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:50.027050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:50.027050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:50.027050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:50.027050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:50.027050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:50.027050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:50.027050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 04:13:50.027050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:50.027050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 415/842] 10.222.0.85 () {56 vars in 1117 bytes} [Tue Aug 30 04:13:49 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:50.037064 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64491c07-3535-4f67-af9a-0fb30686df79 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:50.037318 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 428/843] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:49 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:13:50.072018 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:50.148553 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64491c07-3535-4f67-af9a-0fb30686df79 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:50.155062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8257fc4-34b3-4b7a-bc60-d94e53c25c16 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:50.185476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fb1c3a-2144-4c2e-afb9-28c29363ebc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:50.186091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fb1c3a-2144-4c2e-afb9-28c29363ebc2 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:50.186373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fb1c3a-2144-4c2e-afb9-28c29363ebc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:50.186661 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fb1c3a-2144-4c2e-afb9-28c29363ebc2 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:50.187382 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41fb1c3a-2144-4c2e-afb9-28c29363ebc2 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:50.187685 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41fb1c3a-2144-4c2e-afb9-28c29363ebc2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:50.189503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41fb1c3a-2144-4c2e-afb9-28c29363ebc2 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:50.209167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 416/844] 10.222.0.85 () {56 vars in 1149 bytes} [Tue Aug 30 04:13:50 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:50.216874 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fb1c3a-2144-4c2e-afb9-28c29363ebc2 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:50.222387 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:50.277381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8257fc4-34b3-4b7a-bc60-d94e53c25c16 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:50.315636 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2232f8-b936-487a-b4a0-eea69e6d6002 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:50.316382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2232f8-b936-487a-b4a0-eea69e6d6002 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:50.316667 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2232f8-b936-487a-b4a0-eea69e6d6002 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:50.317005 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2232f8-b936-487a-b4a0-eea69e6d6002 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:50.328602 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2232f8-b936-487a-b4a0-eea69e6d6002 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:50.331031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:50.333609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-9d2232f8-b936-487a-b4a0-eea69e6d6002 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:13:50.333609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:50.333609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:50.333609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:50.333609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:50.333609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:50.333609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:50.333609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:50.333609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:50.333609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:50.333609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:50.333609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:50.333609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:50.333609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:50.333609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:50.333609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:50.334832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:50.334832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:50.334832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:50.334832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:50.334832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:50.334832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:50.334832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:50.334832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:50.334832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:50.334832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:50.334832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:50.334832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:50.334832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:50.334832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:50.334832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:50.334832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:50.336138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:50.336138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:50.336138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:50.336138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:50.336138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:50.336138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:50.336138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:50.336138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:50.336138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:50.336138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:50.336138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:13:50.336138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:50.336138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 429/845] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:13:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:50.336138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2232f8-b936-487a-b4a0-eea69e6d6002 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:50.345623 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fb1c3a-2144-4c2e-afb9-28c29363ebc2 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:50.382622 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5bdca3-ec11-4477-8d85-32fa6688695e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:50.383234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5bdca3-ec11-4477-8d85-32fa6688695e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:50.383524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5bdca3-ec11-4477-8d85-32fa6688695e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:50.383782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5bdca3-ec11-4477-8d85-32fa6688695e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:50.453508 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5bdca3-ec11-4477-8d85-32fa6688695e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:50.454882 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5bdca3-ec11-4477-8d85-32fa6688695e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:50.455095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2232f8-b936-487a-b4a0-eea69e6d6002 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:50.468659 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:50.494543 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3511bf-b5ad-4836-a508-2d929028dd46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:50.495168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3511bf-b5ad-4836-a508-2d929028dd46 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:50.495441 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3511bf-b5ad-4836-a508-2d929028dd46 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:50.495993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3511bf-b5ad-4836-a508-2d929028dd46 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:50.496280 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3511bf-b5ad-4836-a508-2d929028dd46 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:50.498638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3511bf-b5ad-4836-a508-2d929028dd46 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:50.508297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-9e3511bf-b5ad-4836-a508-2d929028dd46 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 04:13:50.508297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:50.508297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:13:50.508297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:13:50.508297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:13:50.508297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:13:50.508297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:13:50.508297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:50.508297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 04:13:50.508297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:50.508297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:13:50.508297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:50.508297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:50.508297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:50.508297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:50.508297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:50.508297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:50.509346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:50.509346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:50.509346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:50.509346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:50.509346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:50.509346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:50.509346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:13:50.509346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:13:50.509346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:13:50.509346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:13:50.509346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:50.509346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:50.509346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:50.509346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:50.509346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:50.509346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:50.514130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:50.514130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:50.514130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:50.514130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:50.514130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:50.514130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:50.514130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:50.514130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:50.514130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:50.514130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:50.514130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:13:50.514130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:50.514130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 04:13:50.514130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:50.514130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 430/846] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:13:50 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:50.514919 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3511bf-b5ad-4836-a508-2d929028dd46 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:50.556663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:50.565912 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 417/847] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:50 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 04:13:50.567303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5bdca3-ec11-4477-8d85-32fa6688695e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:50.636041 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3511bf-b5ad-4836-a508-2d929028dd46 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:50.678188 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd7cef1-564e-4a2b-9ad3-5e5009812282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:50.678797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd7cef1-564e-4a2b-9ad3-5e5009812282 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:50.679230 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd7cef1-564e-4a2b-9ad3-5e5009812282 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:50.679517 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd7cef1-564e-4a2b-9ad3-5e5009812282 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:50.680794 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd7cef1-564e-4a2b-9ad3-5e5009812282 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:50.681221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd7cef1-564e-4a2b-9ad3-5e5009812282 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:50.682506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5bdca3-ec11-4477-8d85-32fa6688695e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:50.682946 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd7cef1-564e-4a2b-9ad3-5e5009812282 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:50.693160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 431/848] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:13:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:50.693997 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd7cef1-564e-4a2b-9ad3-5e5009812282 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:50.725281 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36c5a1e-a5ef-46b6-8962-2c1ed4d15820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:50.725786 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36c5a1e-a5ef-46b6-8962-2c1ed4d15820 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:50.726062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36c5a1e-a5ef-46b6-8962-2c1ed4d15820 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:50.726338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36c5a1e-a5ef-46b6-8962-2c1ed4d15820 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:50.726948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36c5a1e-a5ef-46b6-8962-2c1ed4d15820 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:50.727251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36c5a1e-a5ef-46b6-8962-2c1ed4d15820 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:50.728618 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36c5a1e-a5ef-46b6-8962-2c1ed4d15820 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:50.736450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 418/849] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:13:50 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:50.736450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36c5a1e-a5ef-46b6-8962-2c1ed4d15820 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:50.737755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:50.809027 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:50.819323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd7cef1-564e-4a2b-9ad3-5e5009812282 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:50.854474 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36c5a1e-a5ef-46b6-8962-2c1ed4d15820 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:50.855132 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151bf08a-51e9-434a-b53e-3da5e4d8f72a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:50.855754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151bf08a-51e9-434a-b53e-3da5e4d8f72a None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:50.856028 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151bf08a-51e9-434a-b53e-3da5e4d8f72a None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:50.856294 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151bf08a-51e9-434a-b53e-3da5e4d8f72a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:50.867146 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151bf08a-51e9-434a-b53e-3da5e4d8f72a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:50.868638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151bf08a-51e9-434a-b53e-3da5e4d8f72a None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:50.884980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-151bf08a-51e9-434a-b53e-3da5e4d8f72a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:50.884980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:50.884980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:50.884980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:50.884980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:50.884980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:50.884980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:50.884980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:50.884980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:50.884980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:50.884980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:50.884980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:50.884980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:50.884980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:50.884980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:50.884980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:50.886281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:50.886281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:50.886281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:50.886281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:50.886281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:50.886281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:50.886281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:50.886281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:50.886281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:50.886281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:50.886281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:50.886281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:50.886281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:50.886281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:50.886281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:50.886281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:50.886281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:50.887462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:50.887462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:50.887462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:50.887462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 432/850] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:50.887462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151bf08a-51e9-434a-b53e-3da5e4d8f72a None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:50.898146 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50369bf-b22d-471d-9b55-f5afcbd365a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:50.898799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50369bf-b22d-471d-9b55-f5afcbd365a1 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:50.899081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50369bf-b22d-471d-9b55-f5afcbd365a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:50.899363 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50369bf-b22d-471d-9b55-f5afcbd365a1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:50.910216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50369bf-b22d-471d-9b55-f5afcbd365a1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:50.914054 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50369bf-b22d-471d-9b55-f5afcbd365a1 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:50.930602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-f50369bf-b22d-471d-9b55-f5afcbd365a1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:50.930602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:50.930602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:50.930602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:50.930602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:50.930602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:50.930602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:50.930602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:50.930602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:50.930602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:50.930602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:50.930602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:50.930602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:50.930602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:50.930602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:50.930602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:50.931749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:50.931749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:50.931749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:50.931749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:50.931749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:50.931749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:50.931749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:50.931749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:50.931749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:50.931749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:50.931749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:50.931749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:50.931749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:50.931749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:50.931749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:50.931749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:50.931749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:50.932641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:50.932641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:50.932641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:50.932641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 419/851] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:50.932641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50369bf-b22d-471d-9b55-f5afcbd365a1 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:50.999983 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:51.015767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151bf08a-51e9-434a-b53e-3da5e4d8f72a None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:51.047194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50369bf-b22d-471d-9b55-f5afcbd365a1 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:51.055008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd34d86-9071-408b-8950-2107d5587d96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:51.055634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd34d86-9071-408b-8950-2107d5587d96 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:51.055916 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd34d86-9071-408b-8950-2107d5587d96 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:51.056192 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd34d86-9071-408b-8950-2107d5587d96 None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:51.056836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd34d86-9071-408b-8950-2107d5587d96 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:51.057172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd34d86-9071-408b-8950-2107d5587d96 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:51.058562 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd34d86-9071-408b-8950-2107d5587d96 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:51.059732 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:51.064341 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 433/852] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:13:50 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:51.071160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd34d86-9071-408b-8950-2107d5587d96 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:51.091055 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d86a586-9fa1-4ce1-a06b-fb817c16c875 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:51.091972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d86a586-9fa1-4ce1-a06b-fb817c16c875 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:51.092300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d86a586-9fa1-4ce1-a06b-fb817c16c875 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:51.092717 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d86a586-9fa1-4ce1-a06b-fb817c16c875 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:51.093569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d86a586-9fa1-4ce1-a06b-fb817c16c875 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:51.093978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d86a586-9fa1-4ce1-a06b-fb817c16c875 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:51.095531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d86a586-9fa1-4ce1-a06b-fb817c16c875 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:51.127321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 420/853] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:51.137527 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d86a586-9fa1-4ce1-a06b-fb817c16c875 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:51.223369 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd34d86-9071-408b-8950-2107d5587d96 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:51.267535 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d86a586-9fa1-4ce1-a06b-fb817c16c875 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:51.272694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceaefa69-8897-42a3-a674-53101b43126e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:51.273594 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceaefa69-8897-42a3-a674-53101b43126e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:51.274076 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceaefa69-8897-42a3-a674-53101b43126e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:51.274545 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceaefa69-8897-42a3-a674-53101b43126e None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:51.276492 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceaefa69-8897-42a3-a674-53101b43126e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:51.277152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceaefa69-8897-42a3-a674-53101b43126e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:51.279666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceaefa69-8897-42a3-a674-53101b43126e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:51.296065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 434/854] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:51.303298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceaefa69-8897-42a3-a674-53101b43126e None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:51.313798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2315b5-c190-404c-8acc-e525ffebdcd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:51.314490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2315b5-c190-404c-8acc-e525ffebdcd4 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:51.314879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2315b5-c190-404c-8acc-e525ffebdcd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:51.315165 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2315b5-c190-404c-8acc-e525ffebdcd4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:51.322260 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:51.323385 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2315b5-c190-404c-8acc-e525ffebdcd4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:51.325143 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2315b5-c190-404c-8acc-e525ffebdcd4 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:51.338418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-2f2315b5-c190-404c-8acc-e525ffebdcd4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:51.338418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:51.338418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:51.338418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:51.338418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:51.338418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:51.338418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:51.338418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:51.338418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:51.338418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:51.338418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:51.338418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:51.338418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:51.338418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:51.338418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:51.339374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:51.339374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:51.339374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:51.339374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:51.339374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:51.339374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:51.339374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:51.339374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:51.339374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:51.339374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:51.339374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:51.339374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:51.339374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:51.339374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:51.339374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:51.339374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:51.339374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:51.340279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:51.340279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:51.340279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:51.340279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:51.340279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:51.340279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:51.340279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:51.340279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:51.340279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 421/855] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:51.346218 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2315b5-c190-404c-8acc-e525ffebdcd4 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:51.381704 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:51.427176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceaefa69-8897-42a3-a674-53101b43126e None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:51.466149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffea4c02-b48c-4e93-9f8c-7be6a1a6ca71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:51.466791 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffea4c02-b48c-4e93-9f8c-7be6a1a6ca71 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:51.467073 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffea4c02-b48c-4e93-9f8c-7be6a1a6ca71 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:51.467345 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffea4c02-b48c-4e93-9f8c-7be6a1a6ca71 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:51.471710 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2315b5-c190-404c-8acc-e525ffebdcd4 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:51.475175 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffea4c02-b48c-4e93-9f8c-7be6a1a6ca71 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:51.476631 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffea4c02-b48c-4e93-9f8c-7be6a1a6ca71 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:51.489011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-ffea4c02-b48c-4e93-9f8c-7be6a1a6ca71 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:13:51.489011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:51.489011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:51.489011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:51.489011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:51.489011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:51.489011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:51.489011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:51.489011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:51.489011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:51.489011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:51.489011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:51.489011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:51.489011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:51.489011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:51.489011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:51.490031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:51.490031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:51.490031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:51.490031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:51.490031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:51.490031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:51.490031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:51.490031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:51.490031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:51.490031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:51.490031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:51.490031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:51.490031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:51.490031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:51.490031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:51.490031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:51.490031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:51.490031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:51.490974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:51.490974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:51.490974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:51.490974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:51.490974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:51.490974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:51.490974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:51.490974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:51.490974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 30 04:13:51.490974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:51.490974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 435/856] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:13:51 2022] GET /identity/v3/users/swift => generated 83 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:51.496177 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffea4c02-b48c-4e93-9f8c-7be6a1a6ca71 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:51.529864 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bc76d1-2a24-4abb-b59f-5cf898778bfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:51.531080 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bc76d1-2a24-4abb-b59f-5cf898778bfa None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:51.531992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bc76d1-2a24-4abb-b59f-5cf898778bfa None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:51.532453 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bc76d1-2a24-4abb-b59f-5cf898778bfa None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:51.533648 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bc76d1-2a24-4abb-b59f-5cf898778bfa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:51.534169 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bc76d1-2a24-4abb-b59f-5cf898778bfa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:51.536956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bc76d1-2a24-4abb-b59f-5cf898778bfa None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:51.555068 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 422/857] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:51.561673 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bc76d1-2a24-4abb-b59f-5cf898778bfa None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:51.598650 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:51.640126 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffea4c02-b48c-4e93-9f8c-7be6a1a6ca71 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:51.689725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61f42c4-6260-46c8-8a78-9ca46c672640 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:51.690822 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61f42c4-6260-46c8-8a78-9ca46c672640 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:51.691104 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61f42c4-6260-46c8-8a78-9ca46c672640 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:51.691662 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61f42c4-6260-46c8-8a78-9ca46c672640 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:51.692995 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61f42c4-6260-46c8-8a78-9ca46c672640 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:51.693335 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61f42c4-6260-46c8-8a78-9ca46c672640 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:51.693781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:51.695322 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61f42c4-6260-46c8-8a78-9ca46c672640 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:51.706847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bc76d1-2a24-4abb-b59f-5cf898778bfa None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:51.720975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 436/858] 10.222.0.85 () {56 vars in 1135 bytes} [Tue Aug 30 04:13:51 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:51.727447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61f42c4-6260-46c8-8a78-9ca46c672640 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:51.750838 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd92e6a5-f605-4e50-ac41-eeacf109d1c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:51.751611 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd92e6a5-f605-4e50-ac41-eeacf109d1c8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:51.752227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd92e6a5-f605-4e50-ac41-eeacf109d1c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:51.752672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd92e6a5-f605-4e50-ac41-eeacf109d1c8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:51.753849 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd92e6a5-f605-4e50-ac41-eeacf109d1c8 None admin] RBAC: Adding query filter params (role.id=755eb1bfc0cf445d8f282358f404a320, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=00aeab772672403984bdcb8baaa1520e) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:51.754338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd92e6a5-f605-4e50-ac41-eeacf109d1c8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:51.755943 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd92e6a5-f605-4e50-ac41-eeacf109d1c8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:51.769230 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 423/859] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:13:51 2022] GET /identity/v3/role_assignments?user.id=00aeab772672403984bdcb8baaa1520e&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=755eb1bfc0cf445d8f282358f404a320 => generated 608 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:51.855229 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61f42c4-6260-46c8-8a78-9ca46c672640 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:51.889170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:51.896465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0b218f-4ef8-43de-9914-e07f18ba5ca7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:51.897537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0b218f-4ef8-43de-9914-e07f18ba5ca7 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:51.898379 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0b218f-4ef8-43de-9914-e07f18ba5ca7 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:51.898804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0b218f-4ef8-43de-9914-e07f18ba5ca7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:51.974003 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0b218f-4ef8-43de-9914-e07f18ba5ca7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:51.975862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0b218f-4ef8-43de-9914-e07f18ba5ca7 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:52.083541 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 437/860] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:51 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 04:13:52.089341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd92e6a5-f605-4e50-ac41-eeacf109d1c8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:52.197433 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd92e6a5-f605-4e50-ac41-eeacf109d1c8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:52.237311 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bd8ace-27a1-440d-9314-7884ad96f420 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:52.237963 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bd8ace-27a1-440d-9314-7884ad96f420 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:52.238238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bd8ace-27a1-440d-9314-7884ad96f420 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:52.238506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bd8ace-27a1-440d-9314-7884ad96f420 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:52.249773 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73bd8ace-27a1-440d-9314-7884ad96f420 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:52.251224 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73bd8ace-27a1-440d-9314-7884ad96f420 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:52.267197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-73bd8ace-27a1-440d-9314-7884ad96f420 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:52.267197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:52.267197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:52.267197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:52.267197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:52.267197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:52.267197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:52.267197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:52.267197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:52.267197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:52.267197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:52.267197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:52.267197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:52.267197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:52.267197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:52.267197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:52.268155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:52.268155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:52.268155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:52.268155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:52.268155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:52.268155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:52.268155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:52.268155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:52.268155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:52.268155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:52.268155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:52.268155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:52.268155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:52.268155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:52.268155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:52.268155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:52.268155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:52.269019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:52.269019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:52.269019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:52.269019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 424/861] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:52.274902 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0b218f-4ef8-43de-9914-e07f18ba5ca7 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:52.282538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adaf907-5872-465d-8e66-f5c8ce257c1c None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:52.282824 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adaf907-5872-465d-8e66-f5c8ce257c1c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:52.283205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adaf907-5872-465d-8e66-f5c8ce257c1c None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:52.284530 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 425/862] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:52.289621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45163f2c-6515-4e4f-8d8d-472b3a4b0c0f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:52.289998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45163f2c-6515-4e4f-8d8d-472b3a4b0c0f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:52.290274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45163f2c-6515-4e4f-8d8d-472b3a4b0c0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:52.304452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:52.310658 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:52.311364 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-45163f2c-6515-4e4f-8d8d-472b3a4b0c0f None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:52.312803 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-45163f2c-6515-4e4f-8d8d-472b3a4b0c0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:52.397417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0b218f-4ef8-43de-9914-e07f18ba5ca7 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:52.398332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 426/863] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:52 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:13:52.441553 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26613faf-28f5-4dca-a229-48559c337b0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:52.442714 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26613faf-28f5-4dca-a229-48559c337b0c None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:52.443047 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26613faf-28f5-4dca-a229-48559c337b0c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:52.443321 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26613faf-28f5-4dca-a229-48559c337b0c None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:52.444017 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26613faf-28f5-4dca-a229-48559c337b0c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:52.444324 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26613faf-28f5-4dca-a229-48559c337b0c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:52.446400 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26613faf-28f5-4dca-a229-48559c337b0c None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:52.465928 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 438/864] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:52.471375 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45163f2c-6515-4e4f-8d8d-472b3a4b0c0f None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:52.575429 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45163f2c-6515-4e4f-8d8d-472b3a4b0c0f None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:52.611118 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa4c5fe-b572-47b3-ad6a-3ca0557721a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:52.612137 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa4c5fe-b572-47b3-ad6a-3ca0557721a7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:52.612631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa4c5fe-b572-47b3-ad6a-3ca0557721a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:52.613121 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa4c5fe-b572-47b3-ad6a-3ca0557721a7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:52.616610 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:52.622132 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa4c5fe-b572-47b3-ad6a-3ca0557721a7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:52.623828 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa4c5fe-b572-47b3-ad6a-3ca0557721a7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:52.636302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-8aa4c5fe-b572-47b3-ad6a-3ca0557721a7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:52.636302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:52.636302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:52.636302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:52.636302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:52.636302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:52.636302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:52.636302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:52.636302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:52.636302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:52.636302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:52.636302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:52.636302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:52.636302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:52.636302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:52.637287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:52.637287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:52.637287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:52.637287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:52.637287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:52.637287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:52.637287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:52.637287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:52.637287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:52.637287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:52.637287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:52.637287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:52.637287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:52.637287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:52.637287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:52.637287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:52.637287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:52.638209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:52.638209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:52.638209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:52.638209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:52.638209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:52.638209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:52.638209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:52.638209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:52.640560 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 427/865] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:52.647127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26613faf-28f5-4dca-a229-48559c337b0c None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:52.668358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa4c5fe-b572-47b3-ad6a-3ca0557721a7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:52.771154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:52.780003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa4c5fe-b572-47b3-ad6a-3ca0557721a7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:52.784036 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26613faf-28f5-4dca-a229-48559c337b0c None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:52.814489 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82607d02-d3cb-4525-b993-921d66d87064 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:52.815121 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82607d02-d3cb-4525-b993-921d66d87064 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:52.815402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82607d02-d3cb-4525-b993-921d66d87064 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:52.815677 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82607d02-d3cb-4525-b993-921d66d87064 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:52.816257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82607d02-d3cb-4525-b993-921d66d87064 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:52.818185 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82607d02-d3cb-4525-b993-921d66d87064 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:52.823338 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3d227d-082c-4148-836e-388b38569a88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:52.824199 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3d227d-082c-4148-836e-388b38569a88 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:52.824734 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3d227d-082c-4148-836e-388b38569a88 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:52.825247 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3d227d-082c-4148-836e-388b38569a88 None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:52.826127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3d227d-082c-4148-836e-388b38569a88 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:52.826626 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3d227d-082c-4148-836e-388b38569a88 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:52.827181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-82607d02-d3cb-4525-b993-921d66d87064 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 04:13:52.827181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:52.827181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:13:52.827181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:13:52.827181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:13:52.827181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:13:52.827181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:13:52.827181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:52.827181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 04:13:52.827181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:52.827181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:13:52.827181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:52.827181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:52.827181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:52.827181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:52.827181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:52.828075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:52.828075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:52.828075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:52.828075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:52.828075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:52.828075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:52.828075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:52.828075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:13:52.828075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:13:52.828075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:13:52.828075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:13:52.828075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:52.828075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:52.828075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:52.828075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:52.828075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:52.828075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:52.828973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:52.828973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:52.828973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:52.828973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:52.828973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:52.828973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:52.828973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:52.828973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:52.828973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:52.828973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:52.828973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:13:52.828973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:52.828973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 04:13:52.828973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:52.831381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3d227d-082c-4148-836e-388b38569a88 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:52.833376 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 428/866] 10.222.0.85 () {56 vars in 1132 bytes} [Tue Aug 30 04:13:52 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:52.834842 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82607d02-d3cb-4525-b993-921d66d87064 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:52.846628 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 439/867] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:52.853066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3d227d-082c-4148-836e-388b38569a88 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:52.897689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:52.939132 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82607d02-d3cb-4525-b993-921d66d87064 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:52.968212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3d227d-082c-4148-836e-388b38569a88 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:52.975933 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9595a39-1dc7-4ef0-ae51-71b39ea2d394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:52.976551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9595a39-1dc7-4ef0-ae51-71b39ea2d394 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:52.976832 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9595a39-1dc7-4ef0-ae51-71b39ea2d394 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:52.977134 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9595a39-1dc7-4ef0-ae51-71b39ea2d394 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:52.977772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9595a39-1dc7-4ef0-ae51-71b39ea2d394 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:52.978072 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9595a39-1dc7-4ef0-ae51-71b39ea2d394 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:52.979449 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9595a39-1dc7-4ef0-ae51-71b39ea2d394 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:52.989606 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 429/868] 10.222.0.85 () {56 vars in 1126 bytes} [Tue Aug 30 04:13:52 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:52.991235 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9595a39-1dc7-4ef0-ae51-71b39ea2d394 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:53.009774 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580e010f-c190-47c8-b4ac-1bf4a2b22264 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:53.010776 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580e010f-c190-47c8-b4ac-1bf4a2b22264 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:53.011427 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580e010f-c190-47c8-b4ac-1bf4a2b22264 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:53.011903 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580e010f-c190-47c8-b4ac-1bf4a2b22264 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:53.012965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580e010f-c190-47c8-b4ac-1bf4a2b22264 None admin] RBAC: Adding query filter params (role.id=d17954e7ce544f0d90a512340232c60e, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=6cb539ef6a69406baa70f660e1a815b4) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:53.013561 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580e010f-c190-47c8-b4ac-1bf4a2b22264 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:53.015213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580e010f-c190-47c8-b4ac-1bf4a2b22264 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:53.029241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 440/869] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:13:52 2022] GET /identity/v3/role_assignments?user.id=6cb539ef6a69406baa70f660e1a815b4&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=d17954e7ce544f0d90a512340232c60e => generated 608 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:53.104839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b95845-1aaf-43ec-b178-03e26be59618 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:53.106996 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9595a39-1dc7-4ef0-ae51-71b39ea2d394 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:53.111074 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b95845-1aaf-43ec-b178-03e26be59618 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:53.111690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b95845-1aaf-43ec-b178-03e26be59618 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:53.123958 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 441/870] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:53 2022] GET /identity => generated 270 bytes in 20 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:13:53.125873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bcd20f-dca2-424c-931a-d3eb01442852 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:53.126394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bcd20f-dca2-424c-931a-d3eb01442852 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:53.126878 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bcd20f-dca2-424c-931a-d3eb01442852 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:53.134341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:53.149009 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-08bcd20f-dca2-424c-931a-d3eb01442852 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:53.150265 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-08bcd20f-dca2-424c-931a-d3eb01442852 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:53.151935 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3714bfde-4226-4737-8113-47c97cba478a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:53.152860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3714bfde-4226-4737-8113-47c97cba478a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:53.153429 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3714bfde-4226-4737-8113-47c97cba478a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:53.153769 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3714bfde-4226-4737-8113-47c97cba478a None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:53.154520 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3714bfde-4226-4737-8113-47c97cba478a None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:53.154895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3714bfde-4226-4737-8113-47c97cba478a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:53.156363 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3714bfde-4226-4737-8113-47c97cba478a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:53.162745 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 430/871] 10.222.0.85 () {56 vars in 1126 bytes} [Tue Aug 30 04:13:52 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:53.181444 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:53.238717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 442/872] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:53 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:13:53.513138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3714bfde-4226-4737-8113-47c97cba478a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:53.619684 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3714bfde-4226-4737-8113-47c97cba478a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:53.657678 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d7f159-e5f7-4058-aac9-206fe0eaf3f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:53.658300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d7f159-e5f7-4058-aac9-206fe0eaf3f4 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:53.658583 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d7f159-e5f7-4058-aac9-206fe0eaf3f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:53.658857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d7f159-e5f7-4058-aac9-206fe0eaf3f4 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:53.659527 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d7f159-e5f7-4058-aac9-206fe0eaf3f4 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:53.660939 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d7f159-e5f7-4058-aac9-206fe0eaf3f4 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:53.677972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 431/873] 10.222.0.85 () {60 vars in 1141 bytes} [Tue Aug 30 04:13:53 2022] POST /identity/v3/services => generated 251 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:13:53.972248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0518c0c0-905c-44d8-85ff-01a27c3358a4 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:53.972766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0518c0c0-905c-44d8-85ff-01a27c3358a4 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:53.972844 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0518c0c0-905c-44d8-85ff-01a27c3358a4 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:53.974501 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 443/874] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:53.979730 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4537cc9d-bcd7-48db-bbe2-548591c0abc9 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:53.980154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4537cc9d-bcd7-48db-bbe2-548591c0abc9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:53.980359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4537cc9d-bcd7-48db-bbe2-548591c0abc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:53.999676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4537cc9d-bcd7-48db-bbe2-548591c0abc9 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:54.000879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4537cc9d-bcd7-48db-bbe2-548591c0abc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:54.048847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:54.084701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 432/875] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:53 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:13:54.354201 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0518c0c0-905c-44d8-85ff-01a27c3358a4 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:54.460120 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0518c0c0-905c-44d8-85ff-01a27c3358a4 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:54.496610 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14860965-6d6e-4edc-ad0d-3a270aefb710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:54.497639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14860965-6d6e-4edc-ad0d-3a270aefb710 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:54.498126 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14860965-6d6e-4edc-ad0d-3a270aefb710 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:54.498599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14860965-6d6e-4edc-ad0d-3a270aefb710 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:54.509551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14860965-6d6e-4edc-ad0d-3a270aefb710 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:54.514638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-14860965-6d6e-4edc-ad0d-3a270aefb710 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 04:13:54.514638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:54.514638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:54.514638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:54.514638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:54.514638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:54.514638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:54.514638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:54.514638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:54.514638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:54.514638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:54.514638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:54.514638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:54.514638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:54.514638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:54.514638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:54.515564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:54.515564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:54.515564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:54.515564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:54.515564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:54.515564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:54.515564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:54.515564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:54.515564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:54.515564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:54.515564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:54.515564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:54.515564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:54.515564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:54.515564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:54.515564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:54.516404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:54.516404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:54.516404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:54.516404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:54.516404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:54.516404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:54.516404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:54.516404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:54.516404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:54.516404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:54.516404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 04:13:54.516404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:54.523654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 444/876] 10.222.0.85 () {56 vars in 1120 bytes} [Tue Aug 30 04:13:54 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:54.525271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4537cc9d-bcd7-48db-bbe2-548591c0abc9 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:54.634877 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4537cc9d-bcd7-48db-bbe2-548591c0abc9 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:54.671554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a559fce5-1918-45b2-bdcb-a3cca915a7d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:54.672245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a559fce5-1918-45b2-bdcb-a3cca915a7d5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:54.672531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a559fce5-1918-45b2-bdcb-a3cca915a7d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:54.672879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a559fce5-1918-45b2-bdcb-a3cca915a7d5 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:54.673689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a559fce5-1918-45b2-bdcb-a3cca915a7d5 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:54.673997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a559fce5-1918-45b2-bdcb-a3cca915a7d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:54.675443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a559fce5-1918-45b2-bdcb-a3cca915a7d5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:54.685505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 433/877] 10.222.0.85 () {56 vars in 1115 bytes} [Tue Aug 30 04:13:54 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:54.691324 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14860965-6d6e-4edc-ad0d-3a270aefb710 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:54.697097 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:54.801636 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14860965-6d6e-4edc-ad0d-3a270aefb710 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:54.837868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97860449-9cf9-488b-b531-755376345b5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:54.838538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97860449-9cf9-488b-b531-755376345b5e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:54.838817 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97860449-9cf9-488b-b531-755376345b5e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:54.839086 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97860449-9cf9-488b-b531-755376345b5e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:54.852228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97860449-9cf9-488b-b531-755376345b5e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:54.853822 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97860449-9cf9-488b-b531-755376345b5e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:54.868687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-97860449-9cf9-488b-b531-755376345b5e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:54.868687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:54.868687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:54.868687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:54.868687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:54.868687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:54.868687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:54.868687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:54.868687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:54.868687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:54.868687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:54.868687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:54.868687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:54.868687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:54.868687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:54.868687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:54.869702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:54.869702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:54.869702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:54.869702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:54.869702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:54.869702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:54.869702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:54.869702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:54.869702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:54.869702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:54.869702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:54.869702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:54.869702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:54.869702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:54.869702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:54.869702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:54.869702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:54.870893 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:54.870893 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:54.870893 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:54.870893 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 445/878] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:54.877339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a559fce5-1918-45b2-bdcb-a3cca915a7d5 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:54.915535 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:54.992208 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a559fce5-1918-45b2-bdcb-a3cca915a7d5 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:55.018874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa37f0e-2e3a-4cff-a9c2-51b7a7763366 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:55.019376 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa37f0e-2e3a-4cff-a9c2-51b7a7763366 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:55.019453 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa37f0e-2e3a-4cff-a9c2-51b7a7763366 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:55.021026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 446/879] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:13:55.026103 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4b9ae2-5120-420f-8b1f-d76af76be169 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:55.026486 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4b9ae2-5120-420f-8b1f-d76af76be169 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:55.026756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4b9ae2-5120-420f-8b1f-d76af76be169 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:55.028195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966b29a8-98ca-4afa-94f9-f1e9ee61916b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:55.028813 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966b29a8-98ca-4afa-94f9-f1e9ee61916b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:55.029122 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966b29a8-98ca-4afa-94f9-f1e9ee61916b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:55.029400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966b29a8-98ca-4afa-94f9-f1e9ee61916b None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:55.030116 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966b29a8-98ca-4afa-94f9-f1e9ee61916b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:55.030429 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966b29a8-98ca-4afa-94f9-f1e9ee61916b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:55.031850 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966b29a8-98ca-4afa-94f9-f1e9ee61916b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:55.045458 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-3e4b9ae2-5120-420f-8b1f-d76af76be169 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:55.046407 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 434/880] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:55.046641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-3e4b9ae2-5120-420f-8b1f-d76af76be169 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:55.054790 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966b29a8-98ca-4afa-94f9-f1e9ee61916b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:55.114535 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:55.141405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 447/881] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:55 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:13:55.169322 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332f20e9-e70f-4fdf-abc5-d24b9dff53bf None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:55.169608 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332f20e9-e70f-4fdf-abc5-d24b9dff53bf None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:55.169792 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332f20e9-e70f-4fdf-abc5-d24b9dff53bf None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:55.171481 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 448/882] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:13:55.176361 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938a7cb9-a643-4c26-b3b5-b92aa0beaa49 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:55.176736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938a7cb9-a643-4c26-b3b5-b92aa0beaa49 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:55.177014 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938a7cb9-a643-4c26-b3b5-b92aa0beaa49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:55.183394 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:55.192415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966b29a8-98ca-4afa-94f9-f1e9ee61916b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:55.198042 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-938a7cb9-a643-4c26-b3b5-b92aa0beaa49 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:55.199233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-938a7cb9-a643-4c26-b3b5-b92aa0beaa49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:55.235321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ffa85f-f6e4-491b-b1df-70317e3a3632 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:55.236106 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ffa85f-f6e4-491b-b1df-70317e3a3632 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:55.236607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ffa85f-f6e4-491b-b1df-70317e3a3632 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:55.236951 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ffa85f-f6e4-491b-b1df-70317e3a3632 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:55.246587 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ffa85f-f6e4-491b-b1df-70317e3a3632 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:55.248723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ffa85f-f6e4-491b-b1df-70317e3a3632 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:55.265328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-49ffa85f-f6e4-491b-b1df-70317e3a3632 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 04:13:55.265328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:55.265328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:55.265328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:55.265328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:55.265328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:55.265328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:55.265328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:55.265328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:55.265328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:55.265328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:55.265328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:55.265328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:13:55.265328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:13:55.265328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:13:55.265328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:13:55.266318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:55.266318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:55.266318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:13:55.266318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:55.266318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:13:55.266318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:13:55.266318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:55.266318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:55.266318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:55.266318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:55.266318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:55.266318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:55.266318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:55.266318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:55.266318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:55.266318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:55.266318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:55.266318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:55.267546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:55.267546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:55.267546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:13:55.267546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:13:55.267546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:13:55.267546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:13:55.267546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:13:55.267546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:13:55.267546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 04:13:55.267546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:55.267546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 435/883] 10.222.0.85 () {56 vars in 1117 bytes} [Tue Aug 30 04:13:55 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:55.278059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ffa85f-f6e4-491b-b1df-70317e3a3632 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:55.285312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 449/884] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:55 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:13:55.387291 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ffa85f-f6e4-491b-b1df-70317e3a3632 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:55.422304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45e1f62-87fa-41df-a52c-c3f21a2da483 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:55.422833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45e1f62-87fa-41df-a52c-c3f21a2da483 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:55.423112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45e1f62-87fa-41df-a52c-c3f21a2da483 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:55.423421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45e1f62-87fa-41df-a52c-c3f21a2da483 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:55.424160 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45e1f62-87fa-41df-a52c-c3f21a2da483 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:55.424515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45e1f62-87fa-41df-a52c-c3f21a2da483 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:55.426537 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45e1f62-87fa-41df-a52c-c3f21a2da483 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:55.430687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938a7cb9-a643-4c26-b3b5-b92aa0beaa49 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:55.449081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 436/885] 10.222.0.85 () {56 vars in 1149 bytes} [Tue Aug 30 04:13:55 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:55.452398 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45e1f62-87fa-41df-a52c-c3f21a2da483 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:55.473539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:55.552090 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938a7cb9-a643-4c26-b3b5-b92aa0beaa49 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:55.570683 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:55.584908 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45e1f62-87fa-41df-a52c-c3f21a2da483 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:55.591590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c345be3-b6f8-47aa-a4f1-65601e361b93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:55.592223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c345be3-b6f8-47aa-a4f1-65601e361b93 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:55.593016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c345be3-b6f8-47aa-a4f1-65601e361b93 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:55.593328 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c345be3-b6f8-47aa-a4f1-65601e361b93 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:55.593911 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c345be3-b6f8-47aa-a4f1-65601e361b93 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:55.595626 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c345be3-b6f8-47aa-a4f1-65601e361b93 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:55.605266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-5c345be3-b6f8-47aa-a4f1-65601e361b93 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 04:13:55.605266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:55.605266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:13:55.605266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:13:55.605266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:13:55.605266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:13:55.605266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:13:55.605266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:55.605266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 04:13:55.605266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:55.605266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:13:55.605266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:55.605266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:55.605266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:55.605266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:55.605266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:55.605266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:55.606277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:55.606277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:55.606277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:55.606277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:55.606277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:55.606277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:55.606277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:13:55.606277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:13:55.606277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:13:55.606277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:13:55.606277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:55.606277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:55.606277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:55.606277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:55.606277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:55.606277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:55.607116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:55.607116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:55.607116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:55.607116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:55.607116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:55.607116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:55.607116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:55.607116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:55.607116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:55.607116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:55.607116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:13:55.607116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:55.607116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 04:13:55.607116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:55.607116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 450/886] 10.222.0.85 () {56 vars in 1126 bytes} [Tue Aug 30 04:13:55 2022] GET /identity/v3/services/object-store => generated 93 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:55.607892 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c345be3-b6f8-47aa-a4f1-65601e361b93 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:55.626308 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea68cf83-3b5f-4155-b557-31660613eecf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:55.626928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea68cf83-3b5f-4155-b557-31660613eecf None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:55.627213 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea68cf83-3b5f-4155-b557-31660613eecf None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:55.627484 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea68cf83-3b5f-4155-b557-31660613eecf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:55.711901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea68cf83-3b5f-4155-b557-31660613eecf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:55.713574 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea68cf83-3b5f-4155-b557-31660613eecf None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:55.721333 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c345be3-b6f8-47aa-a4f1-65601e361b93 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:55.729733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:55.757136 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d096f9-cc1e-45aa-ac22-4de299fcd132 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:55.757742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d096f9-cc1e-45aa-ac22-4de299fcd132 None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:55.758025 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d096f9-cc1e-45aa-ac22-4de299fcd132 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:55.758306 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d096f9-cc1e-45aa-ac22-4de299fcd132 None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:55.758955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d096f9-cc1e-45aa-ac22-4de299fcd132 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:55.760531 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d096f9-cc1e-45aa-ac22-4de299fcd132 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:55.776742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 451/887] 10.222.0.85 () {60 vars in 1141 bytes} [Tue Aug 30 04:13:55 2022] POST /identity/v3/services => generated 269 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:13:55.777596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d096f9-cc1e-45aa-ac22-4de299fcd132 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:55.829301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 437/888] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:13:55 2022] GET /identity/v3/auth/tokens => generated 1262 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 04:13:55.831172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdde898-f4fc-44b5-b040-7a6e17f904a9 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:55.831715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdde898-f4fc-44b5-b040-7a6e17f904a9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:55.832190 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdde898-f4fc-44b5-b040-7a6e17f904a9 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:55.834375 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 438/889] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:55.835932 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdde898-f4fc-44b5-b040-7a6e17f904a9 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:55.841174 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:55.920259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d096f9-cc1e-45aa-ac22-4de299fcd132 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:55.960186 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d764bb0-a0d9-40cd-8777-2fe3ffa165e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:55.960772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdde898-f4fc-44b5-b040-7a6e17f904a9 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:55.960911 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d764bb0-a0d9-40cd-8777-2fe3ffa165e4 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:55.961136 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d764bb0-a0d9-40cd-8777-2fe3ffa165e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:55.961412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d764bb0-a0d9-40cd-8777-2fe3ffa165e4 None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:55.962107 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d764bb0-a0d9-40cd-8777-2fe3ffa165e4 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:55.962388 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d764bb0-a0d9-40cd-8777-2fe3ffa165e4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:55.963760 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d764bb0-a0d9-40cd-8777-2fe3ffa165e4 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:55.968515 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 452/890] 10.222.0.85 () {56 vars in 1122 bytes} [Tue Aug 30 04:13:55 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:55.969716 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b0fd50-cf3e-4fa5-8e04-d5e458529871 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:55.969994 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b0fd50-cf3e-4fa5-8e04-d5e458529871 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:55.970260 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b0fd50-cf3e-4fa5-8e04-d5e458529871 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:55.982099 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:55.990882 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-a9b0fd50-cf3e-4fa5-8e04-d5e458529871 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:55.992066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-a9b0fd50-cf3e-4fa5-8e04-d5e458529871 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:56.003428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f281eca1-8731-41f6-bf16-af1697ac5d12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:56.004066 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f281eca1-8731-41f6-bf16-af1697ac5d12 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:56.004862 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f281eca1-8731-41f6-bf16-af1697ac5d12 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:56.005245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f281eca1-8731-41f6-bf16-af1697ac5d12 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:56.016664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f281eca1-8731-41f6-bf16-af1697ac5d12 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:56.018435 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f281eca1-8731-41f6-bf16-af1697ac5d12 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:56.034746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-f281eca1-8731-41f6-bf16-af1697ac5d12 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:56.034746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:56.034746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:56.034746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:56.034746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:56.034746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:56.034746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:56.034746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:56.034746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:56.034746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:56.034746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:56.034746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:56.034746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:56.034746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:56.034746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:56.034746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:56.036276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:56.036276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:56.036276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:56.036276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:56.036276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:56.036276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:56.036276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:56.036276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:56.036276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:56.036276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:56.036276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:56.036276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:56.036276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:56.036276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:56.036276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:56.036276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:56.036276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:56.037637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:56.037637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:56.037637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:56.037637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 439/891] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:56.037637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f281eca1-8731-41f6-bf16-af1697ac5d12 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:56.063964 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 453/892] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:55 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:13:56.065548 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b0fd50-cf3e-4fa5-8e04-d5e458529871 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:56.107772 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:56.166455 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f281eca1-8731-41f6-bf16-af1697ac5d12 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:56.203164 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b0fd50-cf3e-4fa5-8e04-d5e458529871 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:56.210946 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2269374a-5055-443f-af58-19b0230c8ae9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:56.212478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2269374a-5055-443f-af58-19b0230c8ae9 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:56.212962 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2269374a-5055-443f-af58-19b0230c8ae9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:56.213463 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2269374a-5055-443f-af58-19b0230c8ae9 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:56.214739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2269374a-5055-443f-af58-19b0230c8ae9 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:56.215655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2269374a-5055-443f-af58-19b0230c8ae9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:56.217568 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2269374a-5055-443f-af58-19b0230c8ae9 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:56.225079 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 440/893] 10.222.0.85 () {56 vars in 1122 bytes} [Tue Aug 30 04:13:56 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:56.249830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c29615-a166-4421-b896-ae1e3ebafdca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:56.249830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c29615-a166-4421-b896-ae1e3ebafdca None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:56.249830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c29615-a166-4421-b896-ae1e3ebafdca None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:56.249830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c29615-a166-4421-b896-ae1e3ebafdca None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:56.250376 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c29615-a166-4421-b896-ae1e3ebafdca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:56.253066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c29615-a166-4421-b896-ae1e3ebafdca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:56.253066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c29615-a166-4421-b896-ae1e3ebafdca None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:56.265722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 454/894] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:56.272004 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2269374a-5055-443f-af58-19b0230c8ae9 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:56.309634 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:56.374195 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c29615-a166-4421-b896-ae1e3ebafdca None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:56.406167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2269374a-5055-443f-af58-19b0230c8ae9 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:56.443972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1ae7fc-8e9b-4c5b-946e-019e3ebc8cad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:56.444662 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1ae7fc-8e9b-4c5b-946e-019e3ebc8cad None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:56.444947 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1ae7fc-8e9b-4c5b-946e-019e3ebc8cad None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:56.445248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1ae7fc-8e9b-4c5b-946e-019e3ebc8cad None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:56.452302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1ae7fc-8e9b-4c5b-946e-019e3ebc8cad None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:56.453853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1ae7fc-8e9b-4c5b-946e-019e3ebc8cad None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:56.466430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-bb1ae7fc-8e9b-4c5b-946e-019e3ebc8cad None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:56.466430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:56.466430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:56.466430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:56.466430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:56.466430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:56.466430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:56.466430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:56.466430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:56.466430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:56.466430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:56.466430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:56.466430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:13:56.466430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:13:56.466430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:13:56.467324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:13:56.467324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:56.467324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:56.467324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:56.467324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:56.467324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:56.467324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:56.467324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:56.467324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:56.467324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:56.467324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:56.467324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:56.467324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:56.467324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:56.467324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:56.467324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:56.467324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:56.468287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:13:56.468287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:13:56.468287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:13:56.468287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:13:56.468287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:13:56.468287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:13:56.468287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:13:56.468287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:56.468287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 441/895] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:13:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:56.474635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1ae7fc-8e9b-4c5b-946e-019e3ebc8cad None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:56.507328 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:56.520540 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c29615-a166-4421-b896-ae1e3ebafdca None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:56.564398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36359cd-9c9e-48e7-a413-0df85ae24de8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:56.566049 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36359cd-9c9e-48e7-a413-0df85ae24de8 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:56.566540 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36359cd-9c9e-48e7-a413-0df85ae24de8 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:56.567027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36359cd-9c9e-48e7-a413-0df85ae24de8 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:56.568269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36359cd-9c9e-48e7-a413-0df85ae24de8 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:56.570554 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36359cd-9c9e-48e7-a413-0df85ae24de8 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:56.582343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:56.584044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-e36359cd-9c9e-48e7-a413-0df85ae24de8 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 04:13:56.584044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:56.584044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:13:56.584044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:13:56.584044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:13:56.584044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:13:56.584044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:13:56.584044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:56.584044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 04:13:56.584044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:56.584044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:13:56.584044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:56.584044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:56.584044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:56.584044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:56.584044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:56.584044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:56.585011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:56.585011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:56.585011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:56.585011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:56.585011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:56.585011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:56.585011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:13:56.585011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:13:56.585011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:13:56.585011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:13:56.585011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:56.585011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:56.585011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:56.585011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:56.585011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:56.585011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:56.585895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:56.585895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:56.585895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:56.585895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:56.585895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:56.585895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:56.585895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:56.585895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:56.585895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:56.585895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:56.585895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:13:56.585895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:56.585895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 04:13:56.585895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:56.589118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 455/896] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:13:56 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:13:56.592324 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36359cd-9c9e-48e7-a413-0df85ae24de8 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:56.602100 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1ae7fc-8e9b-4c5b-946e-019e3ebc8cad None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:56.639219 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969fd80c-39d6-4f45-9240-b1c1e3db7705 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:56.639831 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969fd80c-39d6-4f45-9240-b1c1e3db7705 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:56.640118 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969fd80c-39d6-4f45-9240-b1c1e3db7705 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:56.641056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969fd80c-39d6-4f45-9240-b1c1e3db7705 None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:56.641190 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969fd80c-39d6-4f45-9240-b1c1e3db7705 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:56.641445 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969fd80c-39d6-4f45-9240-b1c1e3db7705 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:56.643494 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969fd80c-39d6-4f45-9240-b1c1e3db7705 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:56.667455 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 442/897] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:13:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:56.669183 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969fd80c-39d6-4f45-9240-b1c1e3db7705 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:56.721430 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36359cd-9c9e-48e7-a413-0df85ae24de8 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:56.767659 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d58f55-77bd-4323-acd3-f83870142f72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:56.768297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d58f55-77bd-4323-acd3-f83870142f72 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:56.768617 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d58f55-77bd-4323-acd3-f83870142f72 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:56.768891 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d58f55-77bd-4323-acd3-f83870142f72 None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:56.769560 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d58f55-77bd-4323-acd3-f83870142f72 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:56.770507 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d58f55-77bd-4323-acd3-f83870142f72 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:56.771894 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d58f55-77bd-4323-acd3-f83870142f72 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:56.776989 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 456/898] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:13:56 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:56.782908 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d58f55-77bd-4323-acd3-f83870142f72 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:56.797439 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969fd80c-39d6-4f45-9240-b1c1e3db7705 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:56.815241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:56.843776 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864941fa-c4cc-4e17-9e87-f55e9781dde8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:56.844636 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864941fa-c4cc-4e17-9e87-f55e9781dde8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:56.845097 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864941fa-c4cc-4e17-9e87-f55e9781dde8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:56.845996 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864941fa-c4cc-4e17-9e87-f55e9781dde8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:56.846933 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864941fa-c4cc-4e17-9e87-f55e9781dde8 None admin] RBAC: Adding query filter params (role.id=6b76b888c73142da957a10d32ce5e995, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=00aeab772672403984bdcb8baaa1520e) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:56.847393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864941fa-c4cc-4e17-9e87-f55e9781dde8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:56.849757 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864941fa-c4cc-4e17-9e87-f55e9781dde8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:56.865722 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 443/899] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:13:56 2022] GET /identity/v3/role_assignments?user.id=00aeab772672403984bdcb8baaa1520e&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=6b76b888c73142da957a10d32ce5e995 => generated 259 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:56.945175 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d58f55-77bd-4323-acd3-f83870142f72 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:56.993047 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515f64e9-81a4-4845-9cfd-1ca4749569e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:56.993667 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515f64e9-81a4-4845-9cfd-1ca4749569e4 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:56.993947 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515f64e9-81a4-4845-9cfd-1ca4749569e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:56.994220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515f64e9-81a4-4845-9cfd-1ca4749569e4 None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:56.994917 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515f64e9-81a4-4845-9cfd-1ca4749569e4 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:56.995224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515f64e9-81a4-4845-9cfd-1ca4749569e4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:56.996622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515f64e9-81a4-4845-9cfd-1ca4749569e4 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:57.002658 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 457/900] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:13:56 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:57.009693 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864941fa-c4cc-4e17-9e87-f55e9781dde8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:57.027860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:57.146042 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864941fa-c4cc-4e17-9e87-f55e9781dde8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:57.183220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6242003c-7cbf-4703-8259-1a473422a4ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:57.184191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6242003c-7cbf-4703-8259-1a473422a4ec None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:57.184676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6242003c-7cbf-4703-8259-1a473422a4ec None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:57.185227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6242003c-7cbf-4703-8259-1a473422a4ec None admin] PATH_INFO: `/v3/endpoints` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:57.186140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6242003c-7cbf-4703-8259-1a473422a4ec None admin] RBAC: Adding query filter params (interface=public, service_id=7645e6a1a574422b945ceb98b893a94e, region_id=RegionOne) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:57.186651 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6242003c-7cbf-4703-8259-1a473422a4ec None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:57.188248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6242003c-7cbf-4703-8259-1a473422a4ec None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:57.194453 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 444/901] 10.222.0.85 () {56 vars in 1251 bytes} [Tue Aug 30 04:13:57 2022] GET /identity/v3/endpoints?service_id=7645e6a1a574422b945ceb98b893a94e&interface=public®ion_id=RegionOne => generated 194 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:57.200271 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515f64e9-81a4-4845-9cfd-1ca4749569e4 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:57.288592 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:57.308087 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515f64e9-81a4-4845-9cfd-1ca4749569e4 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:57.344611 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7ed13f-293d-47f7-8c63-9c0f62f2b389 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:57.345248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7ed13f-293d-47f7-8c63-9c0f62f2b389 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:57.345532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7ed13f-293d-47f7-8c63-9c0f62f2b389 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:57.345806 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7ed13f-293d-47f7-8c63-9c0f62f2b389 None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:57.346416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7ed13f-293d-47f7-8c63-9c0f62f2b389 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:57.347903 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7ed13f-293d-47f7-8c63-9c0f62f2b389 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:57.354605 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 458/902] 10.222.0.85 () {56 vars in 1087 bytes} [Tue Aug 30 04:13:57 2022] GET /identity/v3/services => generated 1268 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:13:57.799091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ed052e-f8e1-40f3-8bb7-a0dfad279591 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:57.799503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ed052e-f8e1-40f3-8bb7-a0dfad279591 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:57.799949 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ed052e-f8e1-40f3-8bb7-a0dfad279591 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:57.803304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 445/903] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:57 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:13:57.809211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730748c0-5277-4884-959b-62ffaf2df927 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:57.809501 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730748c0-5277-4884-959b-62ffaf2df927 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:57.809762 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730748c0-5277-4884-959b-62ffaf2df927 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:57.831509 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-730748c0-5277-4884-959b-62ffaf2df927 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:57.833143 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-730748c0-5277-4884-959b-62ffaf2df927 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:57.913756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 459/904] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:57 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:13:58.193945 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ed052e-f8e1-40f3-8bb7-a0dfad279591 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:58.241913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb1559f-e783-4e8a-8fb8-76cb8513abe5 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:58.242882 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb1559f-e783-4e8a-8fb8-76cb8513abe5 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:58.243362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb1559f-e783-4e8a-8fb8-76cb8513abe5 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:58.245321 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 460/905] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:13:58.251105 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75364a2-8c74-4119-b818-ec9429343651 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:58.251864 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75364a2-8c74-4119-b818-ec9429343651 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:58.252347 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75364a2-8c74-4119-b818-ec9429343651 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:58.276411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-a75364a2-8c74-4119-b818-ec9429343651 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:58.277627 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-a75364a2-8c74-4119-b818-ec9429343651 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:58.312878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:58.355429 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ed052e-f8e1-40f3-8bb7-a0dfad279591 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:58.367529 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 461/906] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:58 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:13:58.396134 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bd4722-b1de-4db4-9a49-18db05ed6604 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:58.396766 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bd4722-b1de-4db4-9a49-18db05ed6604 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:58.397059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bd4722-b1de-4db4-9a49-18db05ed6604 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:58.397332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bd4722-b1de-4db4-9a49-18db05ed6604 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:58.397915 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bd4722-b1de-4db4-9a49-18db05ed6604 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:58.399271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bd4722-b1de-4db4-9a49-18db05ed6604 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:58.408217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-61bd4722-b1de-4db4-9a49-18db05ed6604 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 04:13:58.408217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:58.408217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:13:58.408217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:13:58.408217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:13:58.408217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:13:58.408217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:13:58.408217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:58.408217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 04:13:58.408217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:58.408217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:13:58.408217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:58.408217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:58.408217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:58.408217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:58.408217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:58.409217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:58.409217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:58.409217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:58.409217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:58.409217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:58.409217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:58.409217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:58.409217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:13:58.409217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:13:58.409217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:13:58.409217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:13:58.409217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:58.409217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:58.409217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:58.409217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:58.409217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:58.409217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:58.410128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:58.410128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:58.410128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:58.410128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:58.410128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:58.410128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:58.410128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:58.410128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:58.410128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:58.410128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:58.410128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:13:58.410128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:13:58.410128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 04:13:58.410128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:58.410128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 446/907] 10.222.0.85 () {56 vars in 1132 bytes} [Tue Aug 30 04:13:58 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:58.418724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75364a2-8c74-4119-b818-ec9429343651 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:58.428037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:58.530237 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75364a2-8c74-4119-b818-ec9429343651 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:58.568334 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9184e1b6-b799-4347-a0ef-2dedc499db1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:58.569437 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9184e1b6-b799-4347-a0ef-2dedc499db1b None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:58.569935 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9184e1b6-b799-4347-a0ef-2dedc499db1b None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:58.570417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9184e1b6-b799-4347-a0ef-2dedc499db1b None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:58.571338 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9184e1b6-b799-4347-a0ef-2dedc499db1b None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:58.571840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9184e1b6-b799-4347-a0ef-2dedc499db1b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:58.573444 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9184e1b6-b799-4347-a0ef-2dedc499db1b None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:58.579035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 462/908] 10.222.0.85 () {56 vars in 1126 bytes} [Tue Aug 30 04:13:58 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:58.586343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bd4722-b1de-4db4-9a49-18db05ed6604 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:58.636414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9184e1b6-b799-4347-a0ef-2dedc499db1b None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:58.704424 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bd4722-b1de-4db4-9a49-18db05ed6604 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:58.735744 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:58.748501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5a5e09-ef4a-4d8a-b58d-644707d7e19a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:58.749170 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5a5e09-ef4a-4d8a-b58d-644707d7e19a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:58.749453 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5a5e09-ef4a-4d8a-b58d-644707d7e19a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:58.749727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5a5e09-ef4a-4d8a-b58d-644707d7e19a None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:58.750758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5a5e09-ef4a-4d8a-b58d-644707d7e19a None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:58.751068 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5a5e09-ef4a-4d8a-b58d-644707d7e19a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:58.752822 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5a5e09-ef4a-4d8a-b58d-644707d7e19a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:58.764430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 447/909] 10.222.0.85 () {56 vars in 1126 bytes} [Tue Aug 30 04:13:58 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:58.765895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5a5e09-ef4a-4d8a-b58d-644707d7e19a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:58.778360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:58.793017 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9184e1b6-b799-4347-a0ef-2dedc499db1b None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:58.841297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beecf8c-5192-4afd-9fd5-f3dd40a6995a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:58.842884 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beecf8c-5192-4afd-9fd5-f3dd40a6995a None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:58.843405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beecf8c-5192-4afd-9fd5-f3dd40a6995a None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:58.844113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beecf8c-5192-4afd-9fd5-f3dd40a6995a None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:58.845267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6beecf8c-5192-4afd-9fd5-f3dd40a6995a None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:58.847511 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6beecf8c-5192-4afd-9fd5-f3dd40a6995a None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:58.868854 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 463/910] 10.222.0.85 () {60 vars in 1141 bytes} [Tue Aug 30 04:13:58 2022] POST /identity/v3/services => generated 241 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:13:58.894936 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5a5e09-ef4a-4d8a-b58d-644707d7e19a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:58.918970 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fde4c9-4b4c-49fc-988c-f8e4777010a0 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:58.919313 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fde4c9-4b4c-49fc-988c-f8e4777010a0 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:58.919510 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fde4c9-4b4c-49fc-988c-f8e4777010a0 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:58.921252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 464/911] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:13:58.927445 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae88b4a-fb39-4d57-abee-bf2f6bc4d1a6 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:58.927773 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae88b4a-fb39-4d57-abee-bf2f6bc4d1a6 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:58.928430 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae88b4a-fb39-4d57-abee-bf2f6bc4d1a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:58.946331 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddda04e-5a0a-4a6f-9f73-3a8b75cad1a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:58.947027 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddda04e-5a0a-4a6f-9f73-3a8b75cad1a7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:58.947308 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddda04e-5a0a-4a6f-9f73-3a8b75cad1a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:58.947579 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddda04e-5a0a-4a6f-9f73-3a8b75cad1a7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:58.948298 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddda04e-5a0a-4a6f-9f73-3a8b75cad1a7 None admin] RBAC: Adding query filter params (interface=public, service_id=4ab3d730491b4cce89efa2cd0c910f23, region_id=RegionOne) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:58.948604 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddda04e-5a0a-4a6f-9f73-3a8b75cad1a7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:58.950091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddda04e-5a0a-4a6f-9f73-3a8b75cad1a7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:58.950752 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-3ae88b4a-fb39-4d57-abee-bf2f6bc4d1a6 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:58.952323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-3ae88b4a-fb39-4d57-abee-bf2f6bc4d1a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:58.957054 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 448/912] 10.222.0.85 () {56 vars in 1251 bytes} [Tue Aug 30 04:13:58 2022] GET /identity/v3/endpoints?service_id=4ab3d730491b4cce89efa2cd0c910f23&interface=public®ion_id=RegionOne => generated 194 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:13:58.962964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddda04e-5a0a-4a6f-9f73-3a8b75cad1a7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:59.052557 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:59.056090 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 465/913] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:58 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:13:59.094517 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddda04e-5a0a-4a6f-9f73-3a8b75cad1a7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:59.138778 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9fb540-a3e2-49be-848e-fff417d13146 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:59.139689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9fb540-a3e2-49be-848e-fff417d13146 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:59.140154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9fb540-a3e2-49be-848e-fff417d13146 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:59.140636 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9fb540-a3e2-49be-848e-fff417d13146 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:59.141465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9fb540-a3e2-49be-848e-fff417d13146 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:59.143085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9fb540-a3e2-49be-848e-fff417d13146 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:59.150736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 449/914] 10.222.0.85 () {56 vars in 1087 bytes} [Tue Aug 30 04:13:58 2022] GET /identity/v3/services => generated 1497 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:13:59.321806 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae88b4a-fb39-4d57-abee-bf2f6bc4d1a6 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:59.385615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:59.433274 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae88b4a-fb39-4d57-abee-bf2f6bc4d1a6 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:59.445716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc3b911-4d38-4f62-a2cf-7672c6125c6d None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:59.446640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc3b911-4d38-4f62-a2cf-7672c6125c6d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:59.447187 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc3b911-4d38-4f62-a2cf-7672c6125c6d None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:59.449005 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 450/915] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:13:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:13:59.454328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84177ec8-2f15-485b-be39-a2af82c8d5d4 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:59.454858 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84177ec8-2f15-485b-be39-a2af82c8d5d4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:59.455368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84177ec8-2f15-485b-be39-a2af82c8d5d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:59.472723 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff874c2-c1cf-4ad4-adfe-3baef0865ac3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:59.473951 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff874c2-c1cf-4ad4-adfe-3baef0865ac3 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:59.474233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff874c2-c1cf-4ad4-adfe-3baef0865ac3 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:59.474511 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff874c2-c1cf-4ad4-adfe-3baef0865ac3 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:59.476102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-84177ec8-2f15-485b-be39-a2af82c8d5d4 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:13:59.477542 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-84177ec8-2f15-485b-be39-a2af82c8d5d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:13:59.491891 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff874c2-c1cf-4ad4-adfe-3baef0865ac3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:13:59.498672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-fff874c2-c1cf-4ad4-adfe-3baef0865ac3 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 04:13:59.498672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:59.498672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:59.498672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:59.498672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:59.498672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:59.498672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:59.498672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:59.498672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:59.498672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:59.498672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:59.498672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:59.498672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:13:59.498672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:13:59.498672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:13:59.498672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:13:59.499579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:13:59.499579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:13:59.499579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:13:59.499579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:13:59.499579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:59.499579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:59.499579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:59.499579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:59.499579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:59.499579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:59.499579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:59.499579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:59.499579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:59.499579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:59.499579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:59.499579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:59.500408 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:59.500408 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:59.500408 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:59.500408 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:59.500408 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:13:59.500408 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:13:59.500408 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:13:59.500408 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:13:59.500408 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:13:59.500408 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:13:59.500408 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 04:13:59.500408 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:13:59.500408 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 466/916] 10.222.0.85 () {56 vars in 1120 bytes} [Tue Aug 30 04:13:59 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:59.508005 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff874c2-c1cf-4ad4-adfe-3baef0865ac3 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:59.573087 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 451/917] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:13:59 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:13:59.615759 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff874c2-c1cf-4ad4-adfe-3baef0865ac3 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:59.640631 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:59.664564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d87ff5-df4e-4c39-8da9-bed76f90308f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:59.665215 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d87ff5-df4e-4c39-8da9-bed76f90308f None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:59.665499 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d87ff5-df4e-4c39-8da9-bed76f90308f None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:59.665832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d87ff5-df4e-4c39-8da9-bed76f90308f None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:59.666507 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d87ff5-df4e-4c39-8da9-bed76f90308f None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:13:59.666860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d87ff5-df4e-4c39-8da9-bed76f90308f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:59.668239 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d87ff5-df4e-4c39-8da9-bed76f90308f None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:59.677680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 467/918] 10.222.0.85 () {56 vars in 1115 bytes} [Tue Aug 30 04:13:59 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:13:59.686196 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84177ec8-2f15-485b-be39-a2af82c8d5d4 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:59.749753 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:13:59.798220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84177ec8-2f15-485b-be39-a2af82c8d5d4 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:59.834426 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38372535-c461-4bc6-b389-f72528776dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:13:59.835062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38372535-c461-4bc6-b389-f72528776dfd None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:13:59.835343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38372535-c461-4bc6-b389-f72528776dfd None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:13:59.835619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38372535-c461-4bc6-b389-f72528776dfd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:13:59.846145 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38372535-c461-4bc6-b389-f72528776dfd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:13:59.847662 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38372535-c461-4bc6-b389-f72528776dfd None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:13:59.862529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-38372535-c461-4bc6-b389-f72528776dfd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:59.862529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:13:59.862529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:13:59.862529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:13:59.862529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:13:59.862529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:13:59.862529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:13:59.862529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:13:59.862529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:13:59.862529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:13:59.862529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:13:59.862529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:13:59.862529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:13:59.862529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:13:59.862529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:13:59.862529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:13:59.863488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:13:59.863488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:13:59.863488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:13:59.863488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:13:59.863488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:13:59.863488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:13:59.863488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:13:59.863488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:13:59.863488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:13:59.863488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:13:59.863488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:13:59.863488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:13:59.863488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:13:59.863488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:13:59.863488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:13:59.863488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:13:59.863488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:13:59.864366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:13:59.864366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:13:59.864366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:13:59.864366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 452/919] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:13:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:13:59.870784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d87ff5-df4e-4c39-8da9-bed76f90308f None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:59.878767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38372535-c461-4bc6-b389-f72528776dfd None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:13:59.980690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d87ff5-df4e-4c39-8da9-bed76f90308f None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:59.984060 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38372535-c461-4bc6-b389-f72528776dfd None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:13:59.997801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:00.019940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71323bb6-2733-48d8-b80c-99cf67666265 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:00.020829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71323bb6-2733-48d8-b80c-99cf67666265 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:00.021297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71323bb6-2733-48d8-b80c-99cf67666265 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:00.021440 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6017ec-38c2-418b-b32c-b04cd87a4f64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:00.021860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71323bb6-2733-48d8-b80c-99cf67666265 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:00.022065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6017ec-38c2-418b-b32c-b04cd87a4f64 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:00.022357 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6017ec-38c2-418b-b32c-b04cd87a4f64 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:00.022692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6017ec-38c2-418b-b32c-b04cd87a4f64 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:00.023176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71323bb6-2733-48d8-b80c-99cf67666265 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:00.023292 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6017ec-38c2-418b-b32c-b04cd87a4f64 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:00.023718 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71323bb6-2733-48d8-b80c-99cf67666265 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:00.024715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6017ec-38c2-418b-b32c-b04cd87a4f64 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:00.025494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71323bb6-2733-48d8-b80c-99cf67666265 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:00.035228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-ee6017ec-38c2-418b-b32c-b04cd87a4f64 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 04:14:00.035228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:00.035228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:14:00.035228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:14:00.035228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:14:00.035228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:14:00.035228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:14:00.035228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:00.035228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 04:14:00.035228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:00.035228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:14:00.035228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:00.035228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:00.035228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:00.035228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:00.035228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:00.035228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:00.036640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:00.036640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:00.036640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:00.036640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:00.036640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:00.036640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:00.036640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:14:00.036640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:14:00.036640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:14:00.036640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:14:00.036640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:00.036640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:00.036640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:00.036640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:00.036640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:00.036640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:00.037774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:00.037774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:00.037774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:00.037774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:00.037774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:00.037774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:00.037774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:00.037774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:00.037774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:00.037774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:00.037774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:14:00.037774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:00.037774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 04:14:00.037774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:00.041409 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 453/920] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:13:59 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:14:00.042592 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6017ec-38c2-418b-b32c-b04cd87a4f64 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:00.043612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 468/921] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:13:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:00.051835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71323bb6-2733-48d8-b80c-99cf67666265 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:00.093687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:00.158076 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6017ec-38c2-418b-b32c-b04cd87a4f64 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:00.169158 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71323bb6-2733-48d8-b80c-99cf67666265 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:00.206032 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfb44b9-2f6f-40bb-a431-387b99c1ece8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:00.206502 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255dfc5d-133b-482b-8d6f-174938a4d2b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:00.206806 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfb44b9-2f6f-40bb-a431-387b99c1ece8 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:00.207174 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfb44b9-2f6f-40bb-a431-387b99c1ece8 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:00.207292 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255dfc5d-133b-482b-8d6f-174938a4d2b3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:00.207464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255dfc5d-133b-482b-8d6f-174938a4d2b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:00.207559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfb44b9-2f6f-40bb-a431-387b99c1ece8 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:00.207733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255dfc5d-133b-482b-8d6f-174938a4d2b3 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:00.208575 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255dfc5d-133b-482b-8d6f-174938a4d2b3 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:00.208884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255dfc5d-133b-482b-8d6f-174938a4d2b3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:00.211177 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255dfc5d-133b-482b-8d6f-174938a4d2b3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:00.215461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfb44b9-2f6f-40bb-a431-387b99c1ece8 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:00.216955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfb44b9-2f6f-40bb-a431-387b99c1ece8 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:00.221118 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 454/922] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:14:00 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:00.222346 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255dfc5d-133b-482b-8d6f-174938a4d2b3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:00.233690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-7cfb44b9-2f6f-40bb-a431-387b99c1ece8 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 04:14:00.233690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:00.233690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:00.233690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:00.233690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:00.233690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:00.233690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:00.233690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:00.233690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:00.233690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:00.233690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:00.233690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:00.233690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:14:00.233690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:14:00.233690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:14:00.233690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:14:00.234614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:00.234614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:00.234614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:14:00.234614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:14:00.234614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:14:00.234614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:14:00.234614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:00.234614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:00.234614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:00.234614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:00.234614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:00.234614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:00.234614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:00.234614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:00.234614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:00.234614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:00.234614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:00.234614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:00.235559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:00.235559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:00.235559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:14:00.235559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:14:00.235559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:14:00.235559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:14:00.235559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:14:00.235559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:14:00.235559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 04:14:00.235559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:00.240018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 469/923] 10.222.0.85 () {56 vars in 1117 bytes} [Tue Aug 30 04:14:00 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:00.241030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfb44b9-2f6f-40bb-a431-387b99c1ece8 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:00.275431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:00.359489 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255dfc5d-133b-482b-8d6f-174938a4d2b3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:00.362657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfb44b9-2f6f-40bb-a431-387b99c1ece8 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:00.364300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:00.396801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad183bb-b3cb-4af9-b602-3a8936792c29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:00.397431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad183bb-b3cb-4af9-b602-3a8936792c29 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:00.397705 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad183bb-b3cb-4af9-b602-3a8936792c29 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:00.397982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad183bb-b3cb-4af9-b602-3a8936792c29 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:00.398697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad183bb-b3cb-4af9-b602-3a8936792c29 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:00.399072 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad183bb-b3cb-4af9-b602-3a8936792c29 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:00.400520 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad183bb-b3cb-4af9-b602-3a8936792c29 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:00.405988 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 455/924] 10.222.0.85 () {56 vars in 1114 bytes} [Tue Aug 30 04:14:00 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:00.411742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7cf103-8251-48d9-9965-5ce360d4c1c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:00.412174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad183bb-b3cb-4af9-b602-3a8936792c29 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:00.412351 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7cf103-8251-48d9-9965-5ce360d4c1c0 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:00.412611 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7cf103-8251-48d9-9965-5ce360d4c1c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:00.412873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7cf103-8251-48d9-9965-5ce360d4c1c0 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:00.413650 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7cf103-8251-48d9-9965-5ce360d4c1c0 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:00.413953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7cf103-8251-48d9-9965-5ce360d4c1c0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:00.415371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7cf103-8251-48d9-9965-5ce360d4c1c0 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:00.435280 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 470/925] 10.222.0.85 () {56 vars in 1149 bytes} [Tue Aug 30 04:14:00 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:00.441840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7cf103-8251-48d9-9965-5ce360d4c1c0 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:00.530611 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad183bb-b3cb-4af9-b602-3a8936792c29 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:00.532892 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:00.553774 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7cf103-8251-48d9-9965-5ce360d4c1c0 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:00.570042 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7189c1fa-f6d3-4b00-9a35-6492341cc3bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:00.570946 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7189c1fa-f6d3-4b00-9a35-6492341cc3bb None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:00.570946 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7189c1fa-f6d3-4b00-9a35-6492341cc3bb None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:00.571214 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7189c1fa-f6d3-4b00-9a35-6492341cc3bb None admin] PATH_INFO: `/v3/endpoints` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:00.571986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7189c1fa-f6d3-4b00-9a35-6492341cc3bb None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:00.573376 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7189c1fa-f6d3-4b00-9a35-6492341cc3bb None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:00.595056 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82aebe59-6aa7-47f8-b4cd-34d1f2743c36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:00.596077 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82aebe59-6aa7-47f8-b4cd-34d1f2743c36 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:00.596591 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82aebe59-6aa7-47f8-b4cd-34d1f2743c36 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:00.597173 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82aebe59-6aa7-47f8-b4cd-34d1f2743c36 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:00.601065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 456/926] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:14:00 2022] POST /identity/v3/endpoints => generated 346 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:14:00.618837 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82aebe59-6aa7-47f8-b4cd-34d1f2743c36 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:00.620723 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82aebe59-6aa7-47f8-b4cd-34d1f2743c36 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:00.640742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-82aebe59-6aa7-47f8-b4cd-34d1f2743c36 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:00.640742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:00.640742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:00.640742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:00.640742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:00.640742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:00.640742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:00.640742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:00.640742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:00.640742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:00.640742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:00.640742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:00.640742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:14:00.640742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:14:00.640742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:14:00.640742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:14:00.641833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:00.641833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:00.641833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:00.641833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:00.641833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:00.641833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:00.641833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:00.641833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:00.641833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:00.641833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:00.641833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:00.641833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:00.641833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:00.641833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:00.641833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:00.641833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:00.641833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:14:00.642708 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:14:00.642708 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:00.642708 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:00.646184 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 471/927] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:00.652134 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7189c1fa-f6d3-4b00-9a35-6492341cc3bb None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:00.778619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7189c1fa-f6d3-4b00-9a35-6492341cc3bb None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:00.818825 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ebf4ab-7711-4ab2-b98e-cfdd9154949f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:00.819370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ebf4ab-7711-4ab2-b98e-cfdd9154949f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:00.819646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ebf4ab-7711-4ab2-b98e-cfdd9154949f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:00.819915 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ebf4ab-7711-4ab2-b98e-cfdd9154949f None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:00.820560 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ebf4ab-7711-4ab2-b98e-cfdd9154949f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:00.820901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ebf4ab-7711-4ab2-b98e-cfdd9154949f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:00.822264 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ebf4ab-7711-4ab2-b98e-cfdd9154949f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:00.836835 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 457/928] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:14:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:00.841774 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82aebe59-6aa7-47f8-b4cd-34d1f2743c36 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:00.858874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:00.887215 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d000f53-3464-4ea1-bda8-c948a87ff34b None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:00.887532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d000f53-3464-4ea1-bda8-c948a87ff34b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:00.887767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d000f53-3464-4ea1-bda8-c948a87ff34b None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:00.889247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 458/929] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:14:00.894273 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6067aa99-12da-4131-b5a4-5cf4d93a63aa None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:00.894853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6067aa99-12da-4131-b5a4-5cf4d93a63aa None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:00.895333 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6067aa99-12da-4131-b5a4-5cf4d93a63aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:00.914676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-6067aa99-12da-4131-b5a4-5cf4d93a63aa None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:00.916452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6067aa99-12da-4131-b5a4-5cf4d93a63aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:00.944546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:00.967886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82aebe59-6aa7-47f8-b4cd-34d1f2743c36 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:01.009089 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 459/930] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:00 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:01.018413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31a40e1-14a5-49ea-b14e-9a53a90d5510 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:01.019864 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31a40e1-14a5-49ea-b14e-9a53a90d5510 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:01.020303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31a40e1-14a5-49ea-b14e-9a53a90d5510 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:01.020884 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31a40e1-14a5-49ea-b14e-9a53a90d5510 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:01.029228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31a40e1-14a5-49ea-b14e-9a53a90d5510 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:01.030895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31a40e1-14a5-49ea-b14e-9a53a90d5510 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:01.043836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-a31a40e1-14a5-49ea-b14e-9a53a90d5510 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:14:01.043836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:01.043836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:01.043836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:01.043836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:01.043836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:01.043836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:01.043836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:01.043836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:01.043836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:01.043836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:01.043836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:01.043836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:14:01.043836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:14:01.043836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:14:01.044956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:14:01.044956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:01.044956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:01.044956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:01.044956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:01.044956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:01.044956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:01.044956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:01.044956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:01.044956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:01.044956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:01.044956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:01.044956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:01.044956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:01.044956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:01.044956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:01.044956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:01.046155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:14:01.046155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:14:01.046155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:14:01.046155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:14:01.046155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:14:01.046155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:14:01.046155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:14:01.046155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:01.053154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 472/931] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:14:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:01.054847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6067aa99-12da-4131-b5a4-5cf4d93a63aa None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:01.174192 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c895c599-c7f9-4e54-8480-3ead4a1d14b8 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:01.174523 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c895c599-c7f9-4e54-8480-3ead4a1d14b8 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:01.174751 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c895c599-c7f9-4e54-8480-3ead4a1d14b8 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:01.176901 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 473/932] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:14:01.181740 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7791763-2477-471f-9f44-228e18b81f16 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:01.182023 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7791763-2477-471f-9f44-228e18b81f16 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:01.182302 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7791763-2477-471f-9f44-228e18b81f16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:01.198025 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6067aa99-12da-4131-b5a4-5cf4d93a63aa None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:01.200845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-a7791763-2477-471f-9f44-228e18b81f16 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:01.202065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-a7791763-2477-471f-9f44-228e18b81f16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:01.244429 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecf4208-2e3a-4056-b48d-5bcd5c1df0df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:01.245059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecf4208-2e3a-4056-b48d-5bcd5c1df0df None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:01.245340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecf4208-2e3a-4056-b48d-5bcd5c1df0df None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:01.245610 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecf4208-2e3a-4056-b48d-5bcd5c1df0df None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:01.246302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecf4208-2e3a-4056-b48d-5bcd5c1df0df None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:01.246617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecf4208-2e3a-4056-b48d-5bcd5c1df0df None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:01.248092 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecf4208-2e3a-4056-b48d-5bcd5c1df0df None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:01.260858 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 460/933] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:14:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:01.267392 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecf4208-2e3a-4056-b48d-5bcd5c1df0df None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:01.279213 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:01.298030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 474/934] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:01 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:14:01.299280 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7791763-2477-471f-9f44-228e18b81f16 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:01.302598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:01.392015 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecf4208-2e3a-4056-b48d-5bcd5c1df0df None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:01.429124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7791763-2477-471f-9f44-228e18b81f16 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:01.435323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70533b5a-43d6-47c9-b417-d9d7d51750fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:01.436220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70533b5a-43d6-47c9-b417-d9d7d51750fa None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:01.436703 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70533b5a-43d6-47c9-b417-d9d7d51750fa None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:01.437238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70533b5a-43d6-47c9-b417-d9d7d51750fa None admin] PATH_INFO: `/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/00aeab772672403984bdcb8baaa1520e/roles/6b76b888c73142da957a10d32ce5e995` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:01.439922 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70533b5a-43d6-47c9-b417-d9d7d51750fa None admin] RBAC: Authorizing `identity:create_grant(project_id=a860dcc3b2d64d1491c40710ec7423a5, user_id=00aeab772672403984bdcb8baaa1520e, role_id=6b76b888c73142da957a10d32ce5e995)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:01.441747 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70533b5a-43d6-47c9-b417-d9d7d51750fa None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:01.461876 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 461/935] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:14:01 2022] PUT /identity/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/00aeab772672403984bdcb8baaa1520e/roles/6b76b888c73142da957a10d32ce5e995 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:14:01.479727 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dc11da-551c-415c-8316-ec16b9ae21ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:01.480835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dc11da-551c-415c-8316-ec16b9ae21ac None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:01.481354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dc11da-551c-415c-8316-ec16b9ae21ac None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:01.481835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dc11da-551c-415c-8316-ec16b9ae21ac None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:01.482645 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6dc11da-551c-415c-8316-ec16b9ae21ac None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:01.484281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6dc11da-551c-415c-8316-ec16b9ae21ac None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:01.495331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-f6dc11da-551c-415c-8316-ec16b9ae21ac None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 04:14:01.495331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:01.495331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:14:01.495331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:14:01.495331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:14:01.495331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:14:01.495331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:14:01.495331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:01.495331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 04:14:01.495331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:01.495331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:14:01.495331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:01.495331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:01.495331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:01.495331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:01.495331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:01.495331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:01.496469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:01.496469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:01.496469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:01.496469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:01.496469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:01.496469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:01.496469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:14:01.496469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:14:01.496469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:14:01.496469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:14:01.496469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:01.496469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:01.496469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:01.496469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:01.496469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:01.496469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:01.497374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:01.497374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:01.497374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:01.497374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:01.497374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:01.497374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:01.497374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:01.497374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:01.497374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:01.497374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:01.497374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:14:01.497374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:01.497374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 04:14:01.497374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:01.500379 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 475/936] 10.222.0.85 () {56 vars in 1126 bytes} [Tue Aug 30 04:14:01 2022] GET /identity/v3/services/object-store => generated 93 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:01.507210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70533b5a-43d6-47c9-b417-d9d7d51750fa None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:01.572372 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dc11da-551c-415c-8316-ec16b9ae21ac None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:01.585005 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:01.627645 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70533b5a-43d6-47c9-b417-d9d7d51750fa None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:01.665426 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:01.675188 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5377fd-8e94-471c-abc2-9cbdc981f9c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:01.676469 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5377fd-8e94-471c-abc2-9cbdc981f9c1 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:01.676911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5377fd-8e94-471c-abc2-9cbdc981f9c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:01.677312 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5377fd-8e94-471c-abc2-9cbdc981f9c1 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:01.678072 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5377fd-8e94-471c-abc2-9cbdc981f9c1 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:01.678379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5377fd-8e94-471c-abc2-9cbdc981f9c1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:01.680348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5377fd-8e94-471c-abc2-9cbdc981f9c1 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:01.687882 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 462/937] 10.222.0.85 () {56 vars in 1122 bytes} [Tue Aug 30 04:14:01 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:01.694423 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5377fd-8e94-471c-abc2-9cbdc981f9c1 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:01.709351 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dc11da-551c-415c-8316-ec16b9ae21ac None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:01.755049 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b573b785-8c36-4edb-9d46-da8cbb774678 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:01.756260 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b573b785-8c36-4edb-9d46-da8cbb774678 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:01.756538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b573b785-8c36-4edb-9d46-da8cbb774678 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:01.756806 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b573b785-8c36-4edb-9d46-da8cbb774678 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:01.757572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b573b785-8c36-4edb-9d46-da8cbb774678 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:01.759261 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b573b785-8c36-4edb-9d46-da8cbb774678 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:01.769530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-b573b785-8c36-4edb-9d46-da8cbb774678 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 04:14:01.769530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:01.769530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:14:01.769530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:14:01.769530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:14:01.769530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:14:01.769530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:14:01.769530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:01.769530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 04:14:01.769530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:01.769530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:14:01.769530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:01.769530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:01.769530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:01.769530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:01.769530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:01.770538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:01.770538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:01.770538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:01.770538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:01.770538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:01.770538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:01.770538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:01.770538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:14:01.770538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:14:01.770538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:14:01.770538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:14:01.770538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:01.770538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:01.770538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:01.770538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:01.770538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:01.770538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:01.771414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:01.771414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:01.771414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:01.771414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:01.771414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:01.771414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:01.771414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:01.771414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:01.771414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:01.771414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:01.771414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:14:01.771414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:01.771414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 04:14:01.771414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:01.771414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 476/938] 10.222.0.85 () {56 vars in 1132 bytes} [Tue Aug 30 04:14:01 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:14:01.779767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b573b785-8c36-4edb-9d46-da8cbb774678 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:01.836083 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5377fd-8e94-471c-abc2-9cbdc981f9c1 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:01.889646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:01.900631 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b573b785-8c36-4edb-9d46-da8cbb774678 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:01.901429 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe6e362-bccc-4455-a80b-b2576d9165d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:01.902615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe6e362-bccc-4455-a80b-b2576d9165d4 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:01.903399 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe6e362-bccc-4455-a80b-b2576d9165d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:01.903992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe6e362-bccc-4455-a80b-b2576d9165d4 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:01.905322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe6e362-bccc-4455-a80b-b2576d9165d4 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:01.906033 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe6e362-bccc-4455-a80b-b2576d9165d4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:01.908481 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe6e362-bccc-4455-a80b-b2576d9165d4 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:01.918209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 463/939] 10.222.0.85 () {56 vars in 1122 bytes} [Tue Aug 30 04:14:01 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:01.930726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe6e362-bccc-4455-a80b-b2576d9165d4 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:01.958758 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:01.966446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63af50b0-8e7a-4259-83db-73412bdc6502 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:01.967273 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63af50b0-8e7a-4259-83db-73412bdc6502 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:01.967409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63af50b0-8e7a-4259-83db-73412bdc6502 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:01.967597 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63af50b0-8e7a-4259-83db-73412bdc6502 None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:01.968234 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63af50b0-8e7a-4259-83db-73412bdc6502 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:01.968535 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63af50b0-8e7a-4259-83db-73412bdc6502 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:01.969945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63af50b0-8e7a-4259-83db-73412bdc6502 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:01.980055 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 477/940] 10.222.0.85 () {56 vars in 1126 bytes} [Tue Aug 30 04:14:01 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:01.981152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63af50b0-8e7a-4259-83db-73412bdc6502 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:02.078704 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe6e362-bccc-4455-a80b-b2576d9165d4 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:02.110767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63af50b0-8e7a-4259-83db-73412bdc6502 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:02.124713 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407228c1-09c5-46ba-a316-80168cf6f929 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:02.125478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407228c1-09c5-46ba-a316-80168cf6f929 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:02.125835 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407228c1-09c5-46ba-a316-80168cf6f929 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:02.126184 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407228c1-09c5-46ba-a316-80168cf6f929 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:02.127324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407228c1-09c5-46ba-a316-80168cf6f929 None admin] RBAC: Adding query filter params (interface=public, service_id=c4b4ec329f1c45ffb6b8f067a9f2b81d, region_id=RegionOne) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:02.127444 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407228c1-09c5-46ba-a316-80168cf6f929 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:02.129146 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407228c1-09c5-46ba-a316-80168cf6f929 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:02.140364 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 464/941] 10.222.0.85 () {56 vars in 1251 bytes} [Tue Aug 30 04:14:01 2022] GET /identity/v3/endpoints?service_id=c4b4ec329f1c45ffb6b8f067a9f2b81d&interface=public®ion_id=RegionOne => generated 194 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:02.141426 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407228c1-09c5-46ba-a316-80168cf6f929 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:02.158382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6d1bea-400e-40cf-b2c7-8d426d630762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:02.159712 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6d1bea-400e-40cf-b2c7-8d426d630762 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:02.161247 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6d1bea-400e-40cf-b2c7-8d426d630762 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:02.161672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6d1bea-400e-40cf-b2c7-8d426d630762 None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:02.162753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6d1bea-400e-40cf-b2c7-8d426d630762 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:02.163204 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6d1bea-400e-40cf-b2c7-8d426d630762 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:02.165405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6d1bea-400e-40cf-b2c7-8d426d630762 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:02.177093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 478/942] 10.222.0.85 () {56 vars in 1126 bytes} [Tue Aug 30 04:14:01 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:02.179518 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6d1bea-400e-40cf-b2c7-8d426d630762 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:02.224266 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:02.259110 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:02.271993 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407228c1-09c5-46ba-a316-80168cf6f929 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:02.313207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cb90ff-fe65-4162-814b-8863e44f1120 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:02.314331 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cb90ff-fe65-4162-814b-8863e44f1120 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:02.314758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cb90ff-fe65-4162-814b-8863e44f1120 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:02.315198 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cb90ff-fe65-4162-814b-8863e44f1120 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:02.315947 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17cb90ff-fe65-4162-814b-8863e44f1120 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:02.316107 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6d1bea-400e-40cf-b2c7-8d426d630762 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:02.318041 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17cb90ff-fe65-4162-814b-8863e44f1120 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:02.326795 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 465/943] 10.222.0.85 () {56 vars in 1087 bytes} [Tue Aug 30 04:14:02 2022] GET /identity/v3/services => generated 1497 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:14:02.364845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e265a32-1b44-41a3-86e8-04ac4757f65b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:02.365843 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e265a32-1b44-41a3-86e8-04ac4757f65b None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:02.366303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e265a32-1b44-41a3-86e8-04ac4757f65b None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:02.366751 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e265a32-1b44-41a3-86e8-04ac4757f65b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:02.367611 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e265a32-1b44-41a3-86e8-04ac4757f65b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:02.369251 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e265a32-1b44-41a3-86e8-04ac4757f65b None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:02.393834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 479/944] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:14:02 2022] POST /identity/v3/endpoints => generated 347 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:14:02.645421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9162e8-e147-4764-8631-ca66618c7f01 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:02.645965 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9162e8-e147-4764-8631-ca66618c7f01 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:02.646252 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9162e8-e147-4764-8631-ca66618c7f01 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:02.648255 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 466/945] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:02.653961 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733b8cc2-cdcf-4a1f-b420-1d6f7ea302b6 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:02.654801 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733b8cc2-cdcf-4a1f-b420-1d6f7ea302b6 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:02.655313 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733b8cc2-cdcf-4a1f-b420-1d6f7ea302b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:02.675032 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-733b8cc2-cdcf-4a1f-b420-1d6f7ea302b6 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:02.677191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-733b8cc2-cdcf-4a1f-b420-1d6f7ea302b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:02.762537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 480/946] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:02 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:03.041335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9162e8-e147-4764-8631-ca66618c7f01 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:03.152752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9162e8-e147-4764-8631-ca66618c7f01 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:03.177025 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:03.190452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e615f7-202c-4665-a965-df563672177f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:03.191074 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e615f7-202c-4665-a965-df563672177f None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:03.191355 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e615f7-202c-4665-a965-df563672177f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:03.191661 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e615f7-202c-4665-a965-df563672177f None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:03.192335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e615f7-202c-4665-a965-df563672177f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:03.193870 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e615f7-202c-4665-a965-df563672177f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:03.196493 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: WARNING keystone.server.flask.common [None req-e5e615f7-202c-4665-a965-df563672177f None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Aug 30 04:14:03.217136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.sql.core [None req-e5e615f7-202c-4665-a965-df563672177f None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 30 04:14:03.217136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 30 04:14:03.217136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [parameters: {'id': '83712040f16a4229a28bdc98e894d434', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 30 04:14:03.217136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81485) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 30 04:14:03.225362 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-e5e615f7-202c-4665-a965-df563672177f None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 83712040f16a4229a28bdc98e894d434.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 83712040f16a4229a28bdc98e894d434. Aug 30 04:14:03.225362 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:03.225362 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 30 04:14:03.225362 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 30 04:14:03.225362 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 30 04:14:03.225362 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 30 04:14:03.225362 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 30 04:14:03.225362 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result = self._query(query) Aug 30 04:14:03.225362 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 30 04:14:03.225362 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application conn.query(q) Aug 30 04:14:03.225362 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 30 04:14:03.225362 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 30 04:14:03.225362 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 30 04:14:03.226447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result.read() Aug 30 04:14:03.226447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 30 04:14:03.226447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 30 04:14:03.226447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 30 04:14:03.226447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 30 04:14:03.226447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 30 04:14:03.226447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 30 04:14:03.226447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 30 04:14:03.226447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 30 04:14:03.226447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 30 04:14:03.226447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:03.226447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 30 04:14:03.226447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:03.226447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:03.226447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 30 04:14:03.226447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 30 04:14:03.226447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Aug 30 04:14:03.226447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return project_ref.to_dict() Aug 30 04:14:03.227823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 30 04:14:03.227823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application next(self.gen) Aug 30 04:14:03.227823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 30 04:14:03.227823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application yield resource Aug 30 04:14:03.227823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 30 04:14:03.227823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application next(self.gen) Aug 30 04:14:03.227823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 30 04:14:03.227823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.session.rollback() Aug 30 04:14:03.227823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 30 04:14:03.227823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.force_reraise() Aug 30 04:14:03.227823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 30 04:14:03.227823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise self.value Aug 30 04:14:03.227823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 30 04:14:03.227823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 30 04:14:03.227823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 30 04:14:03.227823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application session.commit() Aug 30 04:14:03.227823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 30 04:14:03.229149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 30 04:14:03.229149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 30 04:14:03.229149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._prepare_impl() Aug 30 04:14:03.229149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 30 04:14:03.229149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.session.flush() Aug 30 04:14:03.229149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 30 04:14:03.229149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._flush(objects) Aug 30 04:14:03.229149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 30 04:14:03.229149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 30 04:14:03.229149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 30 04:14:03.229149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application compat.raise_( Aug 30 04:14:03.229149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 30 04:14:03.229149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception Aug 30 04:14:03.229149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 30 04:14:03.229149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application flush_context.execute() Aug 30 04:14:03.229149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 30 04:14:03.229149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rec.execute(self) Aug 30 04:14:03.234621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 30 04:14:03.234621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 30 04:14:03.234621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 30 04:14:03.234621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 30 04:14:03.234621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 30 04:14:03.234621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 30 04:14:03.234621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 30 04:14:03.234621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 30 04:14:03.234621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 30 04:14:03.234621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 30 04:14:03.234621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 30 04:14:03.234621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 30 04:14:03.234621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 30 04:14:03.234621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 30 04:14:03.234621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 30 04:14:03.236625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 30 04:14:03.236625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 30 04:14:03.236625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception Aug 30 04:14:03.236625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 30 04:14:03.236625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 30 04:14:03.236625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 30 04:14:03.236625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 30 04:14:03.236625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 30 04:14:03.236625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result = self._query(query) Aug 30 04:14:03.236625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 30 04:14:03.236625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application conn.query(q) Aug 30 04:14:03.236625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 30 04:14:03.236625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 30 04:14:03.236625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 30 04:14:03.236625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result.read() Aug 30 04:14:03.236625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 30 04:14:03.236625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 30 04:14:03.238954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 30 04:14:03.238954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 30 04:14:03.238954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 30 04:14:03.238954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 30 04:14:03.238954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 30 04:14:03.238954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 30 04:14:03.238954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 30 04:14:03.238954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 30 04:14:03.238954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [parameters: {'id': '83712040f16a4229a28bdc98e894d434', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 30 04:14:03.238954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 30 04:14:03.238954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:03.238954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:14:03.238954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:03.238954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:03.238954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Aug 30 04:14:03.240088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Aug 30 04:14:03.240088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 30 04:14:03.240088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 30 04:14:03.240088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Aug 30 04:14:03.240088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:03.240088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:14:03.240088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:03.240088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:03.240088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:03.240088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:03.240088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:03.240088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:03.240088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:03.240088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:03.240088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:03.240088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:03.240088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:03.241402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:03.241402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Aug 30 04:14:03.241402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Aug 30 04:14:03.241402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:03.241402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:03.241402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Aug 30 04:14:03.241402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.Conflict( Aug 30 04:14:03.241402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 83712040f16a4229a28bdc98e894d434. Aug 30 04:14:03.241402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:03.241402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 467/947] 10.222.0.85 () {60 vars in 1140 bytes} [Tue Aug 30 04:14:03 2022] POST /identity/v3/projects => generated 262 bytes in 186 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:03.245340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733b8cc2-cdcf-4a1f-b420-1d6f7ea302b6 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:03.290357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:03.353732 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733b8cc2-cdcf-4a1f-b420-1d6f7ea302b6 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:03.390783 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c173291c-2dd1-4a27-bc6c-b964197976f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:03.391417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c173291c-2dd1-4a27-bc6c-b964197976f7 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:03.391692 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c173291c-2dd1-4a27-bc6c-b964197976f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:03.391959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c173291c-2dd1-4a27-bc6c-b964197976f7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:03.399816 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c173291c-2dd1-4a27-bc6c-b964197976f7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:03.401311 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c173291c-2dd1-4a27-bc6c-b964197976f7 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:03.418323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-c173291c-2dd1-4a27-bc6c-b964197976f7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:14:03.418323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:03.418323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:03.418323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:03.418323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:03.418323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:03.418323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:03.418323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:03.418323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:03.418323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:03.418323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:03.418323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:03.418323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:14:03.418323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:14:03.418323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:14:03.419440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:14:03.419440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:03.419440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:03.419440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:03.419440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:03.419440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:03.419440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:03.419440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:03.419440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:03.419440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:03.419440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:03.419440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:03.419440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:03.419440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:03.419440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:03.419440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:03.419440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:03.420993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:14:03.420993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:14:03.420993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:14:03.420993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:14:03.420993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:14:03.420993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:14:03.420993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:14:03.420993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:03.420993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 481/948] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:14:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:03.428830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e615f7-202c-4665-a965-df563672177f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:03.463002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9c37ca-a307-4a3a-ba5c-95b32d1af195 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:03.463363 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9c37ca-a307-4a3a-ba5c-95b32d1af195 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:03.463542 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9c37ca-a307-4a3a-ba5c-95b32d1af195 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:03.465617 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 482/949] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:14:03.474898 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017aa699-d34c-4f91-aa71-f5fbd5e7e7f8 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:03.475500 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017aa699-d34c-4f91-aa71-f5fbd5e7e7f8 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:03.475997 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017aa699-d34c-4f91-aa71-f5fbd5e7e7f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:03.502555 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-017aa699-d34c-4f91-aa71-f5fbd5e7e7f8 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:03.504973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-017aa699-d34c-4f91-aa71-f5fbd5e7e7f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:03.555692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e615f7-202c-4665-a965-df563672177f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:03.597062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 483/950] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:03 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:14:03.599806 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b955ad65-7b15-4e12-99f1-8bef6d2a315b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:03.600430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b955ad65-7b15-4e12-99f1-8bef6d2a315b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:03.600713 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b955ad65-7b15-4e12-99f1-8bef6d2a315b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:03.600987 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b955ad65-7b15-4e12-99f1-8bef6d2a315b None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:03.601690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b955ad65-7b15-4e12-99f1-8bef6d2a315b None admin] RBAC: Adding query filter params (name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:03.602070 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b955ad65-7b15-4e12-99f1-8bef6d2a315b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:03.603839 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b955ad65-7b15-4e12-99f1-8bef6d2a315b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:03.627558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 468/951] 10.222.0.85 () {56 vars in 1112 bytes} [Tue Aug 30 04:14:03 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:03.862120 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017aa699-d34c-4f91-aa71-f5fbd5e7e7f8 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:03.896029 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:03.970829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017aa699-d34c-4f91-aa71-f5fbd5e7e7f8 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:04.014877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65922da7-f225-4a5a-a3c6-ea8b0a513105 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:04.015561 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65922da7-f225-4a5a-a3c6-ea8b0a513105 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:04.015845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65922da7-f225-4a5a-a3c6-ea8b0a513105 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:04.016112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65922da7-f225-4a5a-a3c6-ea8b0a513105 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:04.027295 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65922da7-f225-4a5a-a3c6-ea8b0a513105 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:14:04.032233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-65922da7-f225-4a5a-a3c6-ea8b0a513105 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 04:14:04.032233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:04.032233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:04.032233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:04.032233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:04.032233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:04.032233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:04.032233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:04.032233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:04.032233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:04.032233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:04.032233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:04.032233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:14:04.032233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:14:04.032233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:14:04.032233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:14:04.033180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:14:04.033180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:14:04.033180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:14:04.033180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:14:04.033180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:04.033180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:04.033180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:04.033180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:04.033180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:04.033180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:04.033180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:04.033180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:04.033180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:04.033180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:04.033180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:04.033180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:04.034027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:04.034027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:04.034027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:04.034027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:04.034027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:14:04.034027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:14:04.034027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:14:04.034027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:14:04.034027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:14:04.034027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:14:04.034027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 04:14:04.034027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:04.034027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 484/952] 10.222.0.85 () {56 vars in 1120 bytes} [Tue Aug 30 04:14:03 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:04.039892 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b955ad65-7b15-4e12-99f1-8bef6d2a315b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:04.090466 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:04.180331 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b955ad65-7b15-4e12-99f1-8bef6d2a315b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:04.215726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70ff44a-0118-49f1-a121-e6d51af4eb63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:04.216346 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70ff44a-0118-49f1-a121-e6d51af4eb63 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:04.216624 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70ff44a-0118-49f1-a121-e6d51af4eb63 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:04.216894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70ff44a-0118-49f1-a121-e6d51af4eb63 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:04.217591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70ff44a-0118-49f1-a121-e6d51af4eb63 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:04.217889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70ff44a-0118-49f1-a121-e6d51af4eb63 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:04.219270 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70ff44a-0118-49f1-a121-e6d51af4eb63 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:04.228732 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 469/953] 10.222.0.85 () {56 vars in 1115 bytes} [Tue Aug 30 04:14:04 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:04.239323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65922da7-f225-4a5a-a3c6-ea8b0a513105 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:04.383104 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:04.393415 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65922da7-f225-4a5a-a3c6-ea8b0a513105 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:04.429454 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec263db3-cae5-49ea-907f-130e313a95b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:04.430086 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec263db3-cae5-49ea-907f-130e313a95b6 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:04.430419 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec263db3-cae5-49ea-907f-130e313a95b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:04.430633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec263db3-cae5-49ea-907f-130e313a95b6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:04.441063 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec263db3-cae5-49ea-907f-130e313a95b6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:04.442551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec263db3-cae5-49ea-907f-130e313a95b6 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:04.458287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-ec263db3-cae5-49ea-907f-130e313a95b6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:04.458287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:04.458287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:04.458287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:04.458287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:04.458287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:04.458287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:04.458287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:04.458287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:04.458287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:04.458287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:04.458287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:04.458287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:14:04.458287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:14:04.458287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:14:04.458287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:14:04.459350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:04.459350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:04.459350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:04.459350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:04.459350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:04.459350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:04.459350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:04.459350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:04.459350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:04.459350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:04.459350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:04.459350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:04.459350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:04.459350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:04.459350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:04.459350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:04.459350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:14:04.460333 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:14:04.460333 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:04.460333 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:04.460333 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 485/954] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:04.467974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70ff44a-0118-49f1-a121-e6d51af4eb63 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:04.588998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:04.592007 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70ff44a-0118-49f1-a121-e6d51af4eb63 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:04.627634 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc03eb8-265d-4400-9329-7a2cc55ee728 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:04.628252 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc03eb8-265d-4400-9329-7a2cc55ee728 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:04.628523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc03eb8-265d-4400-9329-7a2cc55ee728 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:04.628798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc03eb8-265d-4400-9329-7a2cc55ee728 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:04.629538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc03eb8-265d-4400-9329-7a2cc55ee728 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:04.629839 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc03eb8-265d-4400-9329-7a2cc55ee728 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:04.631530 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc03eb8-265d-4400-9329-7a2cc55ee728 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:04.644450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 470/955] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:14:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:04.653683 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec263db3-cae5-49ea-907f-130e313a95b6 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:04.669311 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09020d23-93b7-44df-9204-1d902d062b26 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:04.669582 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09020d23-93b7-44df-9204-1d902d062b26 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:04.670099 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09020d23-93b7-44df-9204-1d902d062b26 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:04.671584 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 471/956] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:14:04.677516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794273e8-e300-4407-86e4-93adf7a82681 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:04.678087 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794273e8-e300-4407-86e4-93adf7a82681 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:04.678087 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794273e8-e300-4407-86e4-93adf7a82681 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:04.697478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-794273e8-e300-4407-86e4-93adf7a82681 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:04.698988 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-794273e8-e300-4407-86e4-93adf7a82681 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:04.777370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 472/957] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:04 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:14:04.778462 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3869059d-69fc-4eca-9aa5-22e69610d169 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:04.778756 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3869059d-69fc-4eca-9aa5-22e69610d169 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:04.779032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3869059d-69fc-4eca-9aa5-22e69610d169 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:04.780399 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 473/958] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:04.786370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f93e8c2-c628-476f-b431-ce29539153f3 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:04.786570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f93e8c2-c628-476f-b431-ce29539153f3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:04.786817 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f93e8c2-c628-476f-b431-ce29539153f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:04.802082 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec263db3-cae5-49ea-907f-130e313a95b6 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:04.806747 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5f93e8c2-c628-476f-b431-ce29539153f3 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:04.808016 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5f93e8c2-c628-476f-b431-ce29539153f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:04.846137 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d7173f-547e-4ff6-92c5-298a1615af06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:04.846884 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d7173f-547e-4ff6-92c5-298a1615af06 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:04.847276 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d7173f-547e-4ff6-92c5-298a1615af06 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:04.847567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d7173f-547e-4ff6-92c5-298a1615af06 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:04.856924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d7173f-547e-4ff6-92c5-298a1615af06 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:04.858651 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d7173f-547e-4ff6-92c5-298a1615af06 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:04.871200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-a6d7173f-547e-4ff6-92c5-298a1615af06 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 04:14:04.871200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:04.871200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:04.871200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:04.871200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:04.871200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:04.871200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:04.871200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:04.871200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:04.871200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:04.871200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:04.871200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:04.871200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:14:04.871200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:14:04.871200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:14:04.871200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:14:04.877910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:04.877910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:04.877910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:14:04.877910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:14:04.877910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:14:04.877910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:14:04.877910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:04.877910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:04.877910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:04.877910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:04.877910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:04.877910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:04.877910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:04.877910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:04.877910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:04.877910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:04.877910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:04.877910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:04.879411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:04.879411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:04.879411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:14:04.879411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:14:04.879411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:14:04.879411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:14:04.879411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:14:04.879411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:14:04.879411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 04:14:04.879411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:04.879411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 486/959] 10.222.0.85 () {56 vars in 1117 bytes} [Tue Aug 30 04:14:04 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:04.879411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d7173f-547e-4ff6-92c5-298a1615af06 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:04.883453 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:04.912143 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 474/960] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:04 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:04.996307 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d7173f-547e-4ff6-92c5-298a1615af06 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:05.050040 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb02ec0-74b9-417a-a85d-3c9274a14bd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:05.050924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb02ec0-74b9-417a-a85d-3c9274a14bd9 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:05.051326 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb02ec0-74b9-417a-a85d-3c9274a14bd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:05.051741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb02ec0-74b9-417a-a85d-3c9274a14bd9 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:05.052803 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb02ec0-74b9-417a-a85d-3c9274a14bd9 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:05.053273 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb02ec0-74b9-417a-a85d-3c9274a14bd9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:05.055530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb02ec0-74b9-417a-a85d-3c9274a14bd9 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:05.056789 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f93e8c2-c628-476f-b431-ce29539153f3 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:05.081441 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 487/961] 10.222.0.85 () {56 vars in 1149 bytes} [Tue Aug 30 04:14:04 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:05.085427 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:05.087774 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb02ec0-74b9-417a-a85d-3c9274a14bd9 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:05.156328 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:05.196548 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f93e8c2-c628-476f-b431-ce29539153f3 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:05.204615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb02ec0-74b9-417a-a85d-3c9274a14bd9 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:05.237389 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8792a4d8-00f5-45ae-b62a-13158af27837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:05.238223 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8792a4d8-00f5-45ae-b62a-13158af27837 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:05.239209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8792a4d8-00f5-45ae-b62a-13158af27837 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:05.239701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8792a4d8-00f5-45ae-b62a-13158af27837 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:05.240551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8792a4d8-00f5-45ae-b62a-13158af27837 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:05.241486 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134fffc3-7d2e-4880-8c0a-d61dfca4cca6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:05.242094 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134fffc3-7d2e-4880-8c0a-d61dfca4cca6 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:05.242371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134fffc3-7d2e-4880-8c0a-d61dfca4cca6 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:05.242641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134fffc3-7d2e-4880-8c0a-d61dfca4cca6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:05.242743 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8792a4d8-00f5-45ae-b62a-13158af27837 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:05.253174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-8792a4d8-00f5-45ae-b62a-13158af27837 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 04:14:05.253174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:05.253174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:14:05.253174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:14:05.253174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:14:05.253174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:14:05.253174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:14:05.253174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:05.253174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 04:14:05.253174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:05.253174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:14:05.253174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:05.253174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:05.253174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:05.253174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:05.253174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:05.253174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:05.254898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:05.254898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:05.254898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:05.254898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:05.254898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:05.254898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:05.254898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:14:05.254898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:14:05.254898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:14:05.254898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:14:05.254898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:05.254898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:05.254898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:05.254898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:05.254898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:05.254898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:05.255761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:05.255761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:05.255761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:05.255761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:05.255761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:05.255761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:05.255761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:05.255761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:05.255761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:05.255761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:05.255761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:14:05.255761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:05.255761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 04:14:05.255761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:05.255761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 475/962] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:14:05 2022] GET /identity/v3/services/compute => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:05.257416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8792a4d8-00f5-45ae-b62a-13158af27837 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:05.325211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134fffc3-7d2e-4880-8c0a-d61dfca4cca6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:05.326661 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134fffc3-7d2e-4880-8c0a-d61dfca4cca6 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:05.362506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:05.385186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8792a4d8-00f5-45ae-b62a-13158af27837 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:05.409548 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:05.424721 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ad105a-13a3-43a1-b1cc-ccf989c4213d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:05.425423 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ad105a-13a3-43a1-b1cc-ccf989c4213d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:05.425707 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ad105a-13a3-43a1-b1cc-ccf989c4213d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:05.426000 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ad105a-13a3-43a1-b1cc-ccf989c4213d None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:05.426591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ad105a-13a3-43a1-b1cc-ccf989c4213d None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:05.427975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ad105a-13a3-43a1-b1cc-ccf989c4213d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:05.436981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-e3ad105a-13a3-43a1-b1cc-ccf989c4213d None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 04:14:05.436981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:05.436981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:14:05.436981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:14:05.436981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:14:05.436981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:14:05.436981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:14:05.436981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:05.436981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 04:14:05.436981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:05.436981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:14:05.436981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:05.436981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:05.436981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:05.436981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:05.436981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:05.436981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:05.438014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:05.438014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:05.438014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:05.438014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:05.438014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:05.438014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:05.438014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:14:05.438014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:14:05.438014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:14:05.438014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:14:05.438014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:05.438014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:05.438014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:05.438014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:05.438014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:05.438014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:05.438855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:05.438855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:05.438855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:05.438855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:05.438855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:05.438855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:05.438855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:05.438855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:05.438855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:05.438855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:05.438855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:14:05.438855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:05.438855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 04:14:05.438855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:05.438855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 476/963] 10.222.0.85 () {56 vars in 1126 bytes} [Tue Aug 30 04:14:05 2022] GET /identity/v3/services/object-store => generated 93 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:05.439682 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ad105a-13a3-43a1-b1cc-ccf989c4213d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:05.465823 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 488/964] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:14:05 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 04:14:05.470104 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134fffc3-7d2e-4880-8c0a-d61dfca4cca6 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:05.577114 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ad105a-13a3-43a1-b1cc-ccf989c4213d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:05.586582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134fffc3-7d2e-4880-8c0a-d61dfca4cca6 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:05.617752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0615b709-1722-40c6-83fa-e9c8bbf06be8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:05.618403 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0615b709-1722-40c6-83fa-e9c8bbf06be8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:05.618704 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0615b709-1722-40c6-83fa-e9c8bbf06be8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:05.618957 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0615b709-1722-40c6-83fa-e9c8bbf06be8 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:05.619599 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0615b709-1722-40c6-83fa-e9c8bbf06be8 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:05.619895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0615b709-1722-40c6-83fa-e9c8bbf06be8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:05.621305 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0615b709-1722-40c6-83fa-e9c8bbf06be8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:05.629781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 477/965] 10.222.0.85 () {56 vars in 1112 bytes} [Tue Aug 30 04:14:05 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:05.629781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0615b709-1722-40c6-83fa-e9c8bbf06be8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:05.638215 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1b2873-2364-49cc-b2c2-e19f8b3f8504 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:05.638830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1b2873-2364-49cc-b2c2-e19f8b3f8504 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:05.639108 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1b2873-2364-49cc-b2c2-e19f8b3f8504 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:05.639377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1b2873-2364-49cc-b2c2-e19f8b3f8504 None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:05.640044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1b2873-2364-49cc-b2c2-e19f8b3f8504 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:05.640394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1b2873-2364-49cc-b2c2-e19f8b3f8504 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:05.641820 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1b2873-2364-49cc-b2c2-e19f8b3f8504 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:05.654736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 489/966] 10.222.0.85 () {56 vars in 1122 bytes} [Tue Aug 30 04:14:05 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:05.654736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1b2873-2364-49cc-b2c2-e19f8b3f8504 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:05.662636 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:05.681264 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:05.765116 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0615b709-1722-40c6-83fa-e9c8bbf06be8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:05.767837 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1b2873-2364-49cc-b2c2-e19f8b3f8504 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:05.802680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96afdee9-de27-4b3f-823e-8f38b74bf6a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:05.803327 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96afdee9-de27-4b3f-823e-8f38b74bf6a6 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:05.803599 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96afdee9-de27-4b3f-823e-8f38b74bf6a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:05.803871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96afdee9-de27-4b3f-823e-8f38b74bf6a6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:05.807477 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30857622-172a-4d1b-b8f8-291a332f02d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:05.808087 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30857622-172a-4d1b-b8f8-291a332f02d4 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:05.808367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30857622-172a-4d1b-b8f8-291a332f02d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:05.808639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30857622-172a-4d1b-b8f8-291a332f02d4 None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:05.809310 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30857622-172a-4d1b-b8f8-291a332f02d4 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:05.809613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30857622-172a-4d1b-b8f8-291a332f02d4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:05.811418 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30857622-172a-4d1b-b8f8-291a332f02d4 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:05.814433 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96afdee9-de27-4b3f-823e-8f38b74bf6a6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:05.815975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96afdee9-de27-4b3f-823e-8f38b74bf6a6 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:05.817459 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 490/967] 10.222.0.85 () {56 vars in 1112 bytes} [Tue Aug 30 04:14:05 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:05.822124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30857622-172a-4d1b-b8f8-291a332f02d4 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:05.842428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-96afdee9-de27-4b3f-823e-8f38b74bf6a6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:05.842428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:05.842428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:05.842428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:05.842428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:05.842428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:05.842428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:05.842428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:05.842428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:05.842428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:05.842428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:05.842428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:05.842428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:14:05.842428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:14:05.842428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:14:05.842428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:14:05.843412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:05.843412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:05.843412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:05.843412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:05.843412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:05.843412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:05.843412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:05.843412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:05.843412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:05.843412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:05.843412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:05.843412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:05.843412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:05.843412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:05.843412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:05.843412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:05.843412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:14:05.844515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:14:05.844515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:05.844515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:05.852331 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 478/968] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:05.853933 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96afdee9-de27-4b3f-823e-8f38b74bf6a6 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:05.937886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:05.946605 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:05.960732 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30857622-172a-4d1b-b8f8-291a332f02d4 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:05.989732 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96afdee9-de27-4b3f-823e-8f38b74bf6a6 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:05.997755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a7c389-8f59-47be-a63b-809ae97e5851 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:05.998367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a7c389-8f59-47be-a63b-809ae97e5851 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:05.998642 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a7c389-8f59-47be-a63b-809ae97e5851 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:05.998913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a7c389-8f59-47be-a63b-809ae97e5851 None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:05.999545 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a7c389-8f59-47be-a63b-809ae97e5851 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:05.999846 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a7c389-8f59-47be-a63b-809ae97e5851 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:06.001233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a7c389-8f59-47be-a63b-809ae97e5851 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:06.007194 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 491/969] 10.222.0.85 () {56 vars in 1122 bytes} [Tue Aug 30 04:14:05 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:06.008370 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e1237d-778a-4daa-9838-f02c5ceea2fc None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:06.008655 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e1237d-778a-4daa-9838-f02c5ceea2fc None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:06.008926 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e1237d-778a-4daa-9838-f02c5ceea2fc None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:06.014082 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 492/970] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:06 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:06.014841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e1237d-778a-4daa-9838-f02c5ceea2fc None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:06.033121 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86108d00-2fde-4cfd-9d33-f3564e88edce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:06.033739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86108d00-2fde-4cfd-9d33-f3564e88edce None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:06.034018 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86108d00-2fde-4cfd-9d33-f3564e88edce None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:06.034291 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86108d00-2fde-4cfd-9d33-f3564e88edce None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:06.034953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86108d00-2fde-4cfd-9d33-f3564e88edce None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:06.035287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86108d00-2fde-4cfd-9d33-f3564e88edce None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:06.036947 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86108d00-2fde-4cfd-9d33-f3564e88edce None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:06.053812 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 479/971] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:14:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:06.053812 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98f6bf8-5d07-46ba-b0ba-a50b384b05d1 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:06.053812 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98f6bf8-5d07-46ba-b0ba-a50b384b05d1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:06.053812 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98f6bf8-5d07-46ba-b0ba-a50b384b05d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:06.069962 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-a98f6bf8-5d07-46ba-b0ba-a50b384b05d1 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:06.071162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a98f6bf8-5d07-46ba-b0ba-a50b384b05d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:06.133912 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e1237d-778a-4daa-9838-f02c5ceea2fc None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:06.148103 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 480/972] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:06 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:06.158133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98f6bf8-5d07-46ba-b0ba-a50b384b05d1 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:06.171223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7cd87e-4dcd-46f7-8506-6e8dd5590040 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:06.172256 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7cd87e-4dcd-46f7-8506-6e8dd5590040 None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:06.172719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7cd87e-4dcd-46f7-8506-6e8dd5590040 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:06.173164 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7cd87e-4dcd-46f7-8506-6e8dd5590040 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:06.174170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7cd87e-4dcd-46f7-8506-6e8dd5590040 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:06.176426 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7cd87e-4dcd-46f7-8506-6e8dd5590040 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:06.218343 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 493/973] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:14:06 2022] POST /identity/v3/endpoints => generated 349 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:14:06.236624 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:06.293753 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98f6bf8-5d07-46ba-b0ba-a50b384b05d1 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:06.342319 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea24d91c-9f8b-4cb9-bf8b-5396ca772946 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:06.343225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea24d91c-9f8b-4cb9-bf8b-5396ca772946 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:06.343695 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea24d91c-9f8b-4cb9-bf8b-5396ca772946 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:06.344154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea24d91c-9f8b-4cb9-bf8b-5396ca772946 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:06.352396 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea24d91c-9f8b-4cb9-bf8b-5396ca772946 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:06.354164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea24d91c-9f8b-4cb9-bf8b-5396ca772946 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:06.366971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-ea24d91c-9f8b-4cb9-bf8b-5396ca772946 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:14:06.366971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:06.366971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:06.366971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:06.366971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:06.366971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:06.366971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:06.366971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:06.366971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:06.366971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:06.366971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:06.366971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:06.366971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:14:06.366971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:14:06.366971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:14:06.368111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:14:06.368111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:06.368111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:06.368111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:06.368111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:06.368111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:06.368111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:06.368111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:06.368111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:06.368111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:06.368111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:06.368111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:06.368111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:06.368111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:06.368111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:06.368111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:06.368111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:06.369240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:14:06.369240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:14:06.369240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:14:06.369240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:14:06.369240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:14:06.369240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:14:06.369240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:14:06.369240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:06.371425 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 481/974] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:14:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:06.377031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7cd87e-4dcd-46f7-8506-6e8dd5590040 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:06.393262 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:06.429373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea24d91c-9f8b-4cb9-bf8b-5396ca772946 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:06.505782 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7cd87e-4dcd-46f7-8506-6e8dd5590040 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:06.547434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c631cdf1-ed9d-437e-a623-9ea3c0b69c8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:06.548191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c631cdf1-ed9d-437e-a623-9ea3c0b69c8d None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:06.548588 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c631cdf1-ed9d-437e-a623-9ea3c0b69c8d None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:06.548953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c631cdf1-ed9d-437e-a623-9ea3c0b69c8d None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:06.549865 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c631cdf1-ed9d-437e-a623-9ea3c0b69c8d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:06.550265 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c631cdf1-ed9d-437e-a623-9ea3c0b69c8d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:06.552688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c631cdf1-ed9d-437e-a623-9ea3c0b69c8d None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:06.561195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea24d91c-9f8b-4cb9-bf8b-5396ca772946 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:06.567034 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 494/975] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:14:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:06.573316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:06.574360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c631cdf1-ed9d-437e-a623-9ea3c0b69c8d None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:06.605699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef0ab11-f118-4dca-a7fc-ed4540fdddde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:06.607019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef0ab11-f118-4dca-a7fc-ed4540fdddde None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:06.607675 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef0ab11-f118-4dca-a7fc-ed4540fdddde None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:06.609382 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef0ab11-f118-4dca-a7fc-ed4540fdddde None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:06.610513 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef0ab11-f118-4dca-a7fc-ed4540fdddde None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:06.612213 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef0ab11-f118-4dca-a7fc-ed4540fdddde None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:06.632815 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.sql.core [None req-bef0ab11-f118-4dca-a7fc-ed4540fdddde None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 30 04:14:06.632815 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 30 04:14:06.632815 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [parameters: {'user_id': '1441119332b24d75aadddcb3eba347bf', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 30 04:14:06.632815 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81485) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 30 04:14:06.641985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-bef0ab11-f118-4dca-a7fc-ed4540fdddde None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 30 04:14:06.641985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:06.641985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 30 04:14:06.641985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 30 04:14:06.641985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 30 04:14:06.641985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 30 04:14:06.641985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 30 04:14:06.641985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result = self._query(query) Aug 30 04:14:06.641985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 30 04:14:06.641985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application conn.query(q) Aug 30 04:14:06.641985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 30 04:14:06.641985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 30 04:14:06.641985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 30 04:14:06.641985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result.read() Aug 30 04:14:06.641985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 30 04:14:06.642930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 30 04:14:06.642930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 30 04:14:06.642930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 30 04:14:06.642930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 30 04:14:06.642930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 30 04:14:06.642930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 30 04:14:06.642930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 30 04:14:06.642930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 30 04:14:06.642930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:06.642930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 30 04:14:06.642930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:06.642930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:06.642930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 30 04:14:06.642930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 30 04:14:06.642930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Aug 30 04:14:06.642930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Aug 30 04:14:06.642930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 30 04:14:06.642930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application next(self.gen) Aug 30 04:14:06.643919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 30 04:14:06.643919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application yield resource Aug 30 04:14:06.643919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 30 04:14:06.643919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application next(self.gen) Aug 30 04:14:06.643919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 30 04:14:06.643919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.session.rollback() Aug 30 04:14:06.643919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 30 04:14:06.643919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.force_reraise() Aug 30 04:14:06.643919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 30 04:14:06.643919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise self.value Aug 30 04:14:06.643919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 30 04:14:06.643919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 30 04:14:06.643919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 30 04:14:06.643919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application session.commit() Aug 30 04:14:06.643919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 30 04:14:06.643919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 30 04:14:06.643919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 30 04:14:06.645115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._prepare_impl() Aug 30 04:14:06.645115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 30 04:14:06.645115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.session.flush() Aug 30 04:14:06.645115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 30 04:14:06.645115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._flush(objects) Aug 30 04:14:06.645115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 30 04:14:06.645115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 30 04:14:06.645115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 30 04:14:06.645115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application compat.raise_( Aug 30 04:14:06.645115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 30 04:14:06.645115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception Aug 30 04:14:06.645115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 30 04:14:06.645115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application flush_context.execute() Aug 30 04:14:06.645115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 30 04:14:06.645115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rec.execute(self) Aug 30 04:14:06.645115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 30 04:14:06.645115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 30 04:14:06.646166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 30 04:14:06.646166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 30 04:14:06.646166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Aug 30 04:14:06.646166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result = connection._execute_20( Aug 30 04:14:06.646166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 30 04:14:06.646166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 30 04:14:06.646166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 30 04:14:06.646166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 30 04:14:06.646166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 30 04:14:06.646166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 30 04:14:06.646166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 30 04:14:06.646166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 30 04:14:06.646166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 30 04:14:06.646166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 30 04:14:06.646166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 30 04:14:06.646166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception Aug 30 04:14:06.648639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 30 04:14:06.648639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 30 04:14:06.648639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 30 04:14:06.648639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 30 04:14:06.648639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 30 04:14:06.648639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result = self._query(query) Aug 30 04:14:06.648639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 30 04:14:06.648639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application conn.query(q) Aug 30 04:14:06.648639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 30 04:14:06.648639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 30 04:14:06.648639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 30 04:14:06.648639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result.read() Aug 30 04:14:06.648639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 30 04:14:06.648639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 30 04:14:06.648639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 30 04:14:06.648639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 30 04:14:06.650788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 30 04:14:06.650788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 30 04:14:06.650788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 30 04:14:06.650788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 30 04:14:06.650788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 30 04:14:06.650788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 30 04:14:06.650788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [parameters: {'user_id': '1441119332b24d75aadddcb3eba347bf', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 30 04:14:06.650788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 30 04:14:06.650788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:06.650788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:14:06.650788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:06.650788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:06.650788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:06.650788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:06.650788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:06.653056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:06.653056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:06.653056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:06.653056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:06.653056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:06.653056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:06.653056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:06.653056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Aug 30 04:14:06.653056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Aug 30 04:14:06.653056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:06.653056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:06.653056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:14:06.653056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:14:06.653056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:14:06.653056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:14:06.653056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Aug 30 04:14:06.653056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Aug 30 04:14:06.656379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Aug 30 04:14:06.656379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Aug 30 04:14:06.656379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 30 04:14:06.656379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 30 04:14:06.656379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 30 04:14:06.656379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:06.656379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 482/976] 10.222.0.85 () {60 vars in 1131 bytes} [Tue Aug 30 04:14:06 2022] POST /identity/v3/users => generated 160 bytes in 214 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:14:06.656379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef0ab11-f118-4dca-a7fc-ed4540fdddde None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:06.671157 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:06.711719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c631cdf1-ed9d-437e-a623-9ea3c0b69c8d None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:06.758031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e3bae0-7474-4c74-8bf2-0144d0e68fa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:06.758654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e3bae0-7474-4c74-8bf2-0144d0e68fa6 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:06.758939 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e3bae0-7474-4c74-8bf2-0144d0e68fa6 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:06.759211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e3bae0-7474-4c74-8bf2-0144d0e68fa6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:06.760386 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e3bae0-7474-4c74-8bf2-0144d0e68fa6 None admin] RBAC: Adding query filter params (role.id=6b76b888c73142da957a10d32ce5e995, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=00aeab772672403984bdcb8baaa1520e) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:06.760699 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e3bae0-7474-4c74-8bf2-0144d0e68fa6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:06.762868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e3bae0-7474-4c74-8bf2-0144d0e68fa6 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:06.772047 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 495/977] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:14:06 2022] GET /identity/v3/role_assignments?user.id=00aeab772672403984bdcb8baaa1520e&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=6b76b888c73142da957a10d32ce5e995 => generated 608 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:06.796855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef0ab11-f118-4dca-a7fc-ed4540fdddde None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:06.839930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a8916d-b484-4b20-abc5-ab6e12f610bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:06.840573 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a8916d-b484-4b20-abc5-ab6e12f610bb None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:06.840855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a8916d-b484-4b20-abc5-ab6e12f610bb None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:06.841151 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a8916d-b484-4b20-abc5-ab6e12f610bb None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:06.849984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a8916d-b484-4b20-abc5-ab6e12f610bb None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:06.851693 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a8916d-b484-4b20-abc5-ab6e12f610bb None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:06.865706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-f3a8916d-b484-4b20-abc5-ab6e12f610bb None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 04:14:06.865706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:06.865706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:06.865706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:06.865706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:06.865706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:06.865706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:06.865706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:06.865706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:06.865706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:06.865706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:06.865706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:06.865706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:14:06.865706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:14:06.865706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:14:06.865706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:14:06.866729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:06.866729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:06.866729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:14:06.866729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:14:06.866729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:14:06.866729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:14:06.866729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:06.866729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:06.866729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:06.866729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:06.866729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:06.866729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:06.866729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:06.866729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:06.866729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:06.866729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:06.866729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:06.866729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:06.867680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:06.867680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:06.867680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:14:06.867680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:14:06.867680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:14:06.867680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:14:06.867680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:14:06.867680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:14:06.867680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 04:14:06.867680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:06.867680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 483/978] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:14:06 2022] GET /identity/v3/users/cinder => generated 84 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:06.874977 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e3bae0-7474-4c74-8bf2-0144d0e68fa6 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:06.994563 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e3bae0-7474-4c74-8bf2-0144d0e68fa6 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:07.026842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:07.032930 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250132af-1125-406d-82c0-e1ec5350df95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:07.033722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250132af-1125-406d-82c0-e1ec5350df95 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:07.034020 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250132af-1125-406d-82c0-e1ec5350df95 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:07.034283 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250132af-1125-406d-82c0-e1ec5350df95 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:07.035039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250132af-1125-406d-82c0-e1ec5350df95 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:07.035872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250132af-1125-406d-82c0-e1ec5350df95 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:07.037323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250132af-1125-406d-82c0-e1ec5350df95 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:07.075215 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 496/979] 10.222.0.85 () {56 vars in 1101 bytes} [Tue Aug 30 04:14:06 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:07.798503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689bf100-5929-4aed-b318-cb30c86210d4 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:07.799187 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689bf100-5929-4aed-b318-cb30c86210d4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:07.799187 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689bf100-5929-4aed-b318-cb30c86210d4 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:07.804165 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 484/980] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:07 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:07.810179 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcda3ec-9075-470f-a1b0-f399471a5699 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:07.810179 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcda3ec-9075-470f-a1b0-f399471a5699 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:07.810415 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcda3ec-9075-470f-a1b0-f399471a5699 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:07.829196 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-7bcda3ec-9075-470f-a1b0-f399471a5699 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:07.830389 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-7bcda3ec-9075-470f-a1b0-f399471a5699 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:07.905559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 497/981] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:07 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:08.178522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689bf100-5929-4aed-b318-cb30c86210d4 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:08.211968 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:08.220033 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9425f387-37fd-4741-8d08-e562fc8ca086 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:08.220554 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9425f387-37fd-4741-8d08-e562fc8ca086 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:08.220909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9425f387-37fd-4741-8d08-e562fc8ca086 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:08.224255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 498/982] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:08 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:08.229615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f60c064-4f01-4ca8-8d85-d963b037fa05 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:08.229903 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f60c064-4f01-4ca8-8d85-d963b037fa05 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:08.230180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f60c064-4f01-4ca8-8d85-d963b037fa05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:08.248633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-4f60c064-4f01-4ca8-8d85-d963b037fa05 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:08.249860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4f60c064-4f01-4ca8-8d85-d963b037fa05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:08.290966 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689bf100-5929-4aed-b318-cb30c86210d4 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:08.325632 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 499/983] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:08 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:08.331610 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6d393c-1a5f-4ee2-933a-f15d084ba5ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:08.332853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6d393c-1a5f-4ee2-933a-f15d084ba5ec None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:08.333253 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6d393c-1a5f-4ee2-933a-f15d084ba5ec None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:08.333620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6d393c-1a5f-4ee2-933a-f15d084ba5ec None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:08.334373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6d393c-1a5f-4ee2-933a-f15d084ba5ec None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:08.335827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6d393c-1a5f-4ee2-933a-f15d084ba5ec None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:08.353108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 485/984] 10.222.0.85 () {60 vars in 1141 bytes} [Tue Aug 30 04:14:08 2022] POST /identity/v3/services => generated 242 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:14:08.593361 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f60c064-4f01-4ca8-8d85-d963b037fa05 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:08.651943 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:08.709905 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f60c064-4f01-4ca8-8d85-d963b037fa05 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:08.752721 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735549e8-facc-42cd-b153-cae4ad8daa97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:08.753550 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735549e8-facc-42cd-b153-cae4ad8daa97 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:08.754023 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735549e8-facc-42cd-b153-cae4ad8daa97 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:08.754388 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735549e8-facc-42cd-b153-cae4ad8daa97 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:08.755061 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735549e8-facc-42cd-b153-cae4ad8daa97 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:08.756779 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735549e8-facc-42cd-b153-cae4ad8daa97 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:08.766398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-735549e8-facc-42cd-b153-cae4ad8daa97 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 04:14:08.766398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:08.766398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:14:08.766398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:14:08.766398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:14:08.766398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:14:08.766398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:14:08.766398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:08.766398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 04:14:08.766398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:08.766398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:14:08.766398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:08.766398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:08.766398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:08.766398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:08.766398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:08.766398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:08.767635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:08.767635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:08.767635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:08.767635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:08.767635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:08.767635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:08.767635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:14:08.767635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:14:08.767635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:14:08.767635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:14:08.767635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:08.767635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:08.767635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:08.767635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:08.767635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:08.767635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:08.768669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:08.768669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:08.768669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:08.768669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:08.768669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:08.768669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:08.768669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:08.768669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:08.768669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:08.768669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:08.768669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:14:08.768669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:08.768669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 04:14:08.768669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:08.770201 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 500/985] 10.222.0.85 () {56 vars in 1126 bytes} [Tue Aug 30 04:14:08 2022] GET /identity/v3/services/object-store => generated 93 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:08.775920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6d393c-1a5f-4ee2-933a-f15d084ba5ec None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:08.803977 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab45d2ba-acbb-4e15-a834-d43c133f81fc None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:08.804268 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab45d2ba-acbb-4e15-a834-d43c133f81fc None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:08.804536 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab45d2ba-acbb-4e15-a834-d43c133f81fc None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:08.806038 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 501/986] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:14:08.811484 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7c91ac-0aee-4c24-b0a1-e210018f7ba5 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:08.811484 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7c91ac-0aee-4c24-b0a1-e210018f7ba5 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:08.811753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7c91ac-0aee-4c24-b0a1-e210018f7ba5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:08.830616 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-5f7c91ac-0aee-4c24-b0a1-e210018f7ba5 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:08.832337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-5f7c91ac-0aee-4c24-b0a1-e210018f7ba5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:08.882015 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:08.893938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6d393c-1a5f-4ee2-933a-f15d084ba5ec None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:08.921094 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 502/987] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:08 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:14:08.934811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cae71e4-ca46-4eb8-ad4c-f8bace3cf97c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:08.935538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cae71e4-ca46-4eb8-ad4c-f8bace3cf97c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:08.935911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cae71e4-ca46-4eb8-ad4c-f8bace3cf97c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:08.936287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cae71e4-ca46-4eb8-ad4c-f8bace3cf97c None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:08.937025 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cae71e4-ca46-4eb8-ad4c-f8bace3cf97c None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:08.937446 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cae71e4-ca46-4eb8-ad4c-f8bace3cf97c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:08.938935 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cae71e4-ca46-4eb8-ad4c-f8bace3cf97c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:08.944627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 486/988] 10.222.0.85 () {56 vars in 1122 bytes} [Tue Aug 30 04:14:08 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:08.950259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7c91ac-0aee-4c24-b0a1-e210018f7ba5 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:08.999720 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:09.076024 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7c91ac-0aee-4c24-b0a1-e210018f7ba5 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:09.125241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b496062-689f-47e5-bdc7-fd9c901dbd71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:09.125998 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b496062-689f-47e5-bdc7-fd9c901dbd71 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:09.126374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b496062-689f-47e5-bdc7-fd9c901dbd71 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:09.126757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b496062-689f-47e5-bdc7-fd9c901dbd71 None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:09.127489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b496062-689f-47e5-bdc7-fd9c901dbd71 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:09.127888 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b496062-689f-47e5-bdc7-fd9c901dbd71 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:09.130034 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b496062-689f-47e5-bdc7-fd9c901dbd71 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:09.141062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 503/989] 10.222.0.85 () {56 vars in 1122 bytes} [Tue Aug 30 04:14:08 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:09.142626 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cae71e4-ca46-4eb8-ad4c-f8bace3cf97c None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:09.224853 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b496062-689f-47e5-bdc7-fd9c901dbd71 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:09.262790 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cae71e4-ca46-4eb8-ad4c-f8bace3cf97c None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:09.299126 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20b2362-05a2-4311-9a9e-a6894763d172 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:09.299766 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20b2362-05a2-4311-9a9e-a6894763d172 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:09.300043 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20b2362-05a2-4311-9a9e-a6894763d172 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:09.300312 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20b2362-05a2-4311-9a9e-a6894763d172 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:09.301639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20b2362-05a2-4311-9a9e-a6894763d172 None admin] RBAC: Adding query filter params (interface=admin, service_id=c4b4ec329f1c45ffb6b8f067a9f2b81d, region_id=RegionOne) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:09.301943 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20b2362-05a2-4311-9a9e-a6894763d172 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:09.303308 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20b2362-05a2-4311-9a9e-a6894763d172 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:09.308735 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 487/990] 10.222.0.85 () {56 vars in 1249 bytes} [Tue Aug 30 04:14:09 2022] GET /identity/v3/endpoints?service_id=c4b4ec329f1c45ffb6b8f067a9f2b81d&interface=admin®ion_id=RegionOne => generated 193 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:09.315519 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20b2362-05a2-4311-9a9e-a6894763d172 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:09.340894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:09.343144 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b496062-689f-47e5-bdc7-fd9c901dbd71 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:09.346821 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:09.384778 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37615f12-44c8-4242-b039-176ff0b580b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:09.385442 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37615f12-44c8-4242-b039-176ff0b580b2 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:09.385733 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37615f12-44c8-4242-b039-176ff0b580b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:09.385985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37615f12-44c8-4242-b039-176ff0b580b2 None admin] PATH_INFO: `/v3/services/image` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:09.386572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37615f12-44c8-4242-b039-176ff0b580b2 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:09.387952 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37615f12-44c8-4242-b039-176ff0b580b2 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:09.398965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-37615f12-44c8-4242-b039-176ff0b580b2 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 04:14:09.398965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:09.398965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:14:09.398965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:14:09.398965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:14:09.398965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:14:09.398965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:14:09.398965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:09.398965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 04:14:09.398965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:09.398965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:14:09.398965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:09.398965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:09.398965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:09.398965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:09.398965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:09.398965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:09.400170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:09.400170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:09.400170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:09.400170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:09.400170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:09.400170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:09.400170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:14:09.400170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:14:09.400170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:14:09.400170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:14:09.400170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:09.400170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:09.400170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:09.400170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:09.400170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:09.400170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:09.401228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:09.401228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:09.401228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:09.401228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:09.401228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:09.401228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:09.401228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:09.401228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:09.401228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:09.401228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:09.401228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:14:09.401228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:09.401228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 04:14:09.401228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:09.403149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 504/991] 10.222.0.85 () {56 vars in 1105 bytes} [Tue Aug 30 04:14:09 2022] GET /identity/v3/services/image => generated 86 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:09.408579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37615f12-44c8-4242-b039-176ff0b580b2 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:09.441892 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20b2362-05a2-4311-9a9e-a6894763d172 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:09.482755 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9d6c7f-72ca-46b4-98bc-95c738538624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:09.483421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9d6c7f-72ca-46b4-98bc-95c738538624 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:09.483702 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9d6c7f-72ca-46b4-98bc-95c738538624 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:09.483973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9d6c7f-72ca-46b4-98bc-95c738538624 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:09.484704 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9d6c7f-72ca-46b4-98bc-95c738538624 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:09.486227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9d6c7f-72ca-46b4-98bc-95c738538624 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:09.505077 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 488/992] 10.222.0.85 () {56 vars in 1087 bytes} [Tue Aug 30 04:14:09 2022] GET /identity/v3/services => generated 1727 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:14:09.533654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37615f12-44c8-4242-b039-176ff0b580b2 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:09.574705 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8c48a0-eeb0-4cb5-b52d-a62828bc70f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:09.575316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8c48a0-eeb0-4cb5-b52d-a62828bc70f8 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:09.575592 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8c48a0-eeb0-4cb5-b52d-a62828bc70f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:09.575861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8c48a0-eeb0-4cb5-b52d-a62828bc70f8 None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:09.576565 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8c48a0-eeb0-4cb5-b52d-a62828bc70f8 None admin] RBAC: Adding query filter params (name=image) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:09.576869 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8c48a0-eeb0-4cb5-b52d-a62828bc70f8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:09.578276 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8c48a0-eeb0-4cb5-b52d-a62828bc70f8 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:09.588558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 505/993] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:09 2022] GET /identity/v3/services?name=image => generated 122 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:09.589982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9d6c7f-72ca-46b4-98bc-95c738538624 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:09.688392 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:09.709562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9d6c7f-72ca-46b4-98bc-95c738538624 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:09.745205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee11a09d-f143-4b21-b30e-90574d74ac21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:09.745916 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee11a09d-f143-4b21-b30e-90574d74ac21 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:09.746258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee11a09d-f143-4b21-b30e-90574d74ac21 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:09.746603 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee11a09d-f143-4b21-b30e-90574d74ac21 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:09.747308 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee11a09d-f143-4b21-b30e-90574d74ac21 None admin] RBAC: Adding query filter params (type=image) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:09.747684 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee11a09d-f143-4b21-b30e-90574d74ac21 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:09.749235 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee11a09d-f143-4b21-b30e-90574d74ac21 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:09.761070 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 489/994] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:09 2022] GET /identity/v3/services?type=image => generated 122 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:10.310900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a791ed-58b6-4e2d-8287-d3e036a4d115 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:10.311420 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a791ed-58b6-4e2d-8287-d3e036a4d115 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:10.311420 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a791ed-58b6-4e2d-8287-d3e036a4d115 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:10.312906 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 506/995] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:10.318422 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7f74aa-c64d-4696-9fbc-a05f5488c729 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:10.318862 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7f74aa-c64d-4696-9fbc-a05f5488c729 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:10.318956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7f74aa-c64d-4696-9fbc-a05f5488c729 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:10.340186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-fc7f74aa-c64d-4696-9fbc-a05f5488c729 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:10.341719 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fc7f74aa-c64d-4696-9fbc-a05f5488c729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:10.411743 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 490/996] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:10 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:10.679050 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a791ed-58b6-4e2d-8287-d3e036a4d115 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:10.714401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:10.797401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a791ed-58b6-4e2d-8287-d3e036a4d115 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:10.838884 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ce4308-a829-41ab-beb5-d82225ac023b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:10.839534 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ce4308-a829-41ab-beb5-d82225ac023b None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:10.839809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ce4308-a829-41ab-beb5-d82225ac023b None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:10.840078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ce4308-a829-41ab-beb5-d82225ac023b None admin] PATH_INFO: `/v3/services/compute` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:10.840668 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ce4308-a829-41ab-beb5-d82225ac023b None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:10.842085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ce4308-a829-41ab-beb5-d82225ac023b None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:10.852130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-b1ce4308-a829-41ab-beb5-d82225ac023b None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 04:14:10.852130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:10.852130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:14:10.852130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:14:10.852130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:14:10.852130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:14:10.852130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:14:10.852130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:10.852130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 04:14:10.852130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:10.852130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:14:10.852130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:10.852130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:10.852130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:10.852130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:10.852130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:10.852130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:10.853205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:10.853205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:10.853205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:10.853205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:10.853205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:10.853205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:10.853205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:14:10.853205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:14:10.853205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:14:10.853205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:14:10.853205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:10.853205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:10.853205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:10.853205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:10.853205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:10.853205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:10.854263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:10.854263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:10.854263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:10.854263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:10.854263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:10.854263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:10.854263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:10.854263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:10.854263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:10.854263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:10.854263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:14:10.854263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:10.854263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 04:14:10.854263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:10.857346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 507/997] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:14:10 2022] GET /identity/v3/services/compute => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:10.859459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7f74aa-c64d-4696-9fbc-a05f5488c729 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:10.943473 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:10.966282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7f74aa-c64d-4696-9fbc-a05f5488c729 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:11.001145 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9076038-b59f-4f38-b325-cdcf2c106436 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:11.001817 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9076038-b59f-4f38-b325-cdcf2c106436 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:11.002108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9076038-b59f-4f38-b325-cdcf2c106436 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:11.002378 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9076038-b59f-4f38-b325-cdcf2c106436 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:11.003033 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9076038-b59f-4f38-b325-cdcf2c106436 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:11.003368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9076038-b59f-4f38-b325-cdcf2c106436 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:11.004797 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9076038-b59f-4f38-b325-cdcf2c106436 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:11.010232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 491/998] 10.222.0.85 () {56 vars in 1112 bytes} [Tue Aug 30 04:14:10 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:11.015649 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ce4308-a829-41ab-beb5-d82225ac023b None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:11.121939 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:11.130041 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ce4308-a829-41ab-beb5-d82225ac023b None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:11.167678 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0870b35e-f1f8-4bfb-8335-14a563953bfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:11.168291 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0870b35e-f1f8-4bfb-8335-14a563953bfe None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:11.168758 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0870b35e-f1f8-4bfb-8335-14a563953bfe None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:11.168957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0870b35e-f1f8-4bfb-8335-14a563953bfe None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:11.169551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0870b35e-f1f8-4bfb-8335-14a563953bfe None admin] RBAC: Adding query filter params (type=compute) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:11.169991 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0870b35e-f1f8-4bfb-8335-14a563953bfe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:11.171352 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0870b35e-f1f8-4bfb-8335-14a563953bfe None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:11.185195 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 508/999] 10.222.0.85 () {56 vars in 1112 bytes} [Tue Aug 30 04:14:11 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:11.189070 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9076038-b59f-4f38-b325-cdcf2c106436 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:11.308213 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9076038-b59f-4f38-b325-cdcf2c106436 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:11.352948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0152a1a6-75c1-443b-a04d-ad777b1ef588 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:11.353624 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0152a1a6-75c1-443b-a04d-ad777b1ef588 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:11.353916 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0152a1a6-75c1-443b-a04d-ad777b1ef588 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:11.354194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0152a1a6-75c1-443b-a04d-ad777b1ef588 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:11.354902 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0152a1a6-75c1-443b-a04d-ad777b1ef588 None admin] RBAC: Adding query filter params (interface=public, service_id=30e9a31a8fc342d7895cb95a7edd935a, region_id=RegionOne) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:11.355210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0152a1a6-75c1-443b-a04d-ad777b1ef588 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:11.356592 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0152a1a6-75c1-443b-a04d-ad777b1ef588 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:11.365256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 492/1000] 10.222.0.85 () {56 vars in 1251 bytes} [Tue Aug 30 04:14:11 2022] GET /identity/v3/endpoints?service_id=30e9a31a8fc342d7895cb95a7edd935a&interface=public®ion_id=RegionOne => generated 194 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:11.368919 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0870b35e-f1f8-4bfb-8335-14a563953bfe None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:11.443809 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2d2926-9169-4ba8-8672-1f027aca5646 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:11.444161 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2d2926-9169-4ba8-8672-1f027aca5646 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:11.444471 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2d2926-9169-4ba8-8672-1f027aca5646 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:11.446001 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 493/1001] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:11.451035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868c5a34-4707-4672-91c8-a8219d9a1368 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:11.451323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868c5a34-4707-4672-91c8-a8219d9a1368 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:11.451618 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868c5a34-4707-4672-91c8-a8219d9a1368 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:11.465502 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:11.473840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-868c5a34-4707-4672-91c8-a8219d9a1368 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:11.475059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-868c5a34-4707-4672-91c8-a8219d9a1368 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:11.478147 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0870b35e-f1f8-4bfb-8335-14a563953bfe None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:11.518106 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa3e478-c967-46cd-bb14-0b251c0d97cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:11.519443 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa3e478-c967-46cd-bb14-0b251c0d97cb None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:11.519443 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa3e478-c967-46cd-bb14-0b251c0d97cb None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:11.519443 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa3e478-c967-46cd-bb14-0b251c0d97cb None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:11.520598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa3e478-c967-46cd-bb14-0b251c0d97cb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:11.521619 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa3e478-c967-46cd-bb14-0b251c0d97cb None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:11.528692 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 509/1002] 10.222.0.85 () {56 vars in 1087 bytes} [Tue Aug 30 04:14:11 2022] GET /identity/v3/services => generated 1727 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:14:11.551117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 494/1003] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:11 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:14:11.686694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b64a5e-96e3-4960-995f-9f8137de186d None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:11.687105 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b64a5e-96e3-4960-995f-9f8137de186d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:11.687242 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b64a5e-96e3-4960-995f-9f8137de186d None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:11.688579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 510/1004] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:11.694002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac8307c-3265-475c-a02a-7149c5d6734f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:11.694284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac8307c-3265-475c-a02a-7149c5d6734f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:11.694575 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac8307c-3265-475c-a02a-7149c5d6734f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:11.714279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-cac8307c-3265-475c-a02a-7149c5d6734f None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:11.715486 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cac8307c-3265-475c-a02a-7149c5d6734f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:11.793092 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 495/1005] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:11 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:11.826168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b64a5e-96e3-4960-995f-9f8137de186d None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:11.854152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:11.942037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b64a5e-96e3-4960-995f-9f8137de186d None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:11.978378 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8c0e05-b9c9-4e3a-b84a-a4e95cfc2615 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:11.979010 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8c0e05-b9c9-4e3a-b84a-a4e95cfc2615 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:11.979290 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8c0e05-b9c9-4e3a-b84a-a4e95cfc2615 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:11.979552 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8c0e05-b9c9-4e3a-b84a-a4e95cfc2615 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:11.980132 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8c0e05-b9c9-4e3a-b84a-a4e95cfc2615 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:11.981530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8c0e05-b9c9-4e3a-b84a-a4e95cfc2615 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:11.991766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-fa8c0e05-b9c9-4e3a-b84a-a4e95cfc2615 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 04:14:11.991766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:11.991766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:14:11.991766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:14:11.991766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:14:11.991766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:14:11.991766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:14:11.991766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:11.991766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 04:14:11.991766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:11.991766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:14:11.991766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:11.991766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:11.991766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:11.991766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:11.991766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:11.991766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:11.992797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:11.992797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:11.992797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:11.992797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:11.992797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:11.992797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:11.992797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:14:11.992797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:14:11.992797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:14:11.992797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:14:11.992797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:11.992797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:11.992797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:11.992797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:11.992797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:11.992797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:11.993704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:11.993704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:11.993704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:11.993704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:11.993704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:11.993704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:11.993704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:11.993704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:11.993704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:11.993704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:11.993704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:14:11.993704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:11.993704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 04:14:11.993704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:11.997404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 511/1006] 10.222.0.85 () {56 vars in 1126 bytes} [Tue Aug 30 04:14:11 2022] GET /identity/v3/services/object-store => generated 93 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:11.998783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac8307c-3265-475c-a02a-7149c5d6734f None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:12.063184 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8c0e05-b9c9-4e3a-b84a-a4e95cfc2615 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:12.068573 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:12.116509 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac8307c-3265-475c-a02a-7149c5d6734f None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:12.157480 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2540fd-7fd3-41d6-a4dc-5d7c25495e24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:12.158739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2540fd-7fd3-41d6-a4dc-5d7c25495e24 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:12.159379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2540fd-7fd3-41d6-a4dc-5d7c25495e24 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:12.159842 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2540fd-7fd3-41d6-a4dc-5d7c25495e24 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:12.161521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2540fd-7fd3-41d6-a4dc-5d7c25495e24 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:12.161975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2540fd-7fd3-41d6-a4dc-5d7c25495e24 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:12.164377 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2540fd-7fd3-41d6-a4dc-5d7c25495e24 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:12.168244 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:12.173304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 496/1007] 10.222.0.85 () {56 vars in 1122 bytes} [Tue Aug 30 04:14:11 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:12.178697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2540fd-7fd3-41d6-a4dc-5d7c25495e24 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:12.189015 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8c0e05-b9c9-4e3a-b84a-a4e95cfc2615 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:12.228964 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bddd018-591b-498a-8e10-e81ee54f98de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:12.229522 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bddd018-591b-498a-8e10-e81ee54f98de None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:12.229810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bddd018-591b-498a-8e10-e81ee54f98de None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:12.230082 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bddd018-591b-498a-8e10-e81ee54f98de None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:12.230844 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bddd018-591b-498a-8e10-e81ee54f98de None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:12.232115 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bddd018-591b-498a-8e10-e81ee54f98de None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:12.249315 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 512/1008] 10.222.0.85 () {60 vars in 1141 bytes} [Tue Aug 30 04:14:12 2022] POST /identity/v3/services => generated 242 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:14:12.299254 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2540fd-7fd3-41d6-a4dc-5d7c25495e24 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:12.335924 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:12.349713 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6647f81-8714-433d-bce9-c22a471e80a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:12.350321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6647f81-8714-433d-bce9-c22a471e80a8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:12.350600 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6647f81-8714-433d-bce9-c22a471e80a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:12.350884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6647f81-8714-433d-bce9-c22a471e80a8 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:12.351514 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6647f81-8714-433d-bce9-c22a471e80a8 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:12.351883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6647f81-8714-433d-bce9-c22a471e80a8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:12.353219 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6647f81-8714-433d-bce9-c22a471e80a8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:12.363782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 497/1009] 10.222.0.85 () {56 vars in 1122 bytes} [Tue Aug 30 04:14:12 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:12.364927 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bddd018-591b-498a-8e10-e81ee54f98de None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:12.489515 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bddd018-591b-498a-8e10-e81ee54f98de None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:12.526950 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1364174-e9bd-48ab-a633-d14080bb8c65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:12.527566 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1364174-e9bd-48ab-a633-d14080bb8c65 None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:12.527846 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1364174-e9bd-48ab-a633-d14080bb8c65 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:12.528119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1364174-e9bd-48ab-a633-d14080bb8c65 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:12.528812 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1364174-e9bd-48ab-a633-d14080bb8c65 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:12.530222 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1364174-e9bd-48ab-a633-d14080bb8c65 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:12.548354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 513/1010] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:14:12 2022] POST /identity/v3/endpoints => generated 325 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:14:13.508671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c9503e-081a-4fb4-a1a6-9fc3a152b04a None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:13.509256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c9503e-081a-4fb4-a1a6-9fc3a152b04a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:13.509256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c9503e-081a-4fb4-a1a6-9fc3a152b04a None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:13.510635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 498/1011] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:13.516005 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b353af3-a8cd-445b-b5e2-b009820eb98e None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:13.516554 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b353af3-a8cd-445b-b5e2-b009820eb98e None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:13.516554 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b353af3-a8cd-445b-b5e2-b009820eb98e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:13.540653 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-1b353af3-a8cd-445b-b5e2-b009820eb98e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:13.541491 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-1b353af3-a8cd-445b-b5e2-b009820eb98e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:13.543868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:13.623373 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 514/1012] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:13 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:13.897091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c9503e-081a-4fb4-a1a6-9fc3a152b04a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:14.004726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c9503e-081a-4fb4-a1a6-9fc3a152b04a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:14.042086 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83526a6-35e5-46d6-84e3-ecdb17ff65cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:14.042731 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83526a6-35e5-46d6-84e3-ecdb17ff65cd None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:14.043011 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83526a6-35e5-46d6-84e3-ecdb17ff65cd None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:14.043284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83526a6-35e5-46d6-84e3-ecdb17ff65cd None admin] PATH_INFO: `/v3/services/compute` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:14.043867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83526a6-35e5-46d6-84e3-ecdb17ff65cd None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:14.045269 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83526a6-35e5-46d6-84e3-ecdb17ff65cd None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:14.055194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-d83526a6-35e5-46d6-84e3-ecdb17ff65cd None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 04:14:14.055194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:14.055194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:14:14.055194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:14:14.055194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:14:14.055194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:14:14.055194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:14:14.055194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:14.055194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 04:14:14.055194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:14.055194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:14:14.055194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:14.055194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:14.055194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:14.055194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:14.055194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:14.055194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:14.056225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:14.056225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:14.056225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:14.056225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:14.056225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:14.056225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:14.056225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:14:14.056225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:14:14.056225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:14:14.056225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:14:14.056225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:14.056225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:14.056225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:14.056225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:14.056225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:14.056225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:14.057668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:14.057668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:14.057668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:14.057668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:14.057668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:14.057668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:14.057668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:14.057668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:14.057668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:14.057668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:14.057668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:14:14.057668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:14.057668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 04:14:14.057668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:14.061407 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 499/1013] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:14:13 2022] GET /identity/v3/services/compute => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:14:14.062278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b353af3-a8cd-445b-b5e2-b009820eb98e None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:14.176964 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b353af3-a8cd-445b-b5e2-b009820eb98e None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:14.213141 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5073e06c-cc5d-4139-8e9e-786fabbbad11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:14.213757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5073e06c-cc5d-4139-8e9e-786fabbbad11 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:14.214052 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5073e06c-cc5d-4139-8e9e-786fabbbad11 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:14.214301 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5073e06c-cc5d-4139-8e9e-786fabbbad11 None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:14.214968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5073e06c-cc5d-4139-8e9e-786fabbbad11 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:14.215239 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5073e06c-cc5d-4139-8e9e-786fabbbad11 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:14.216604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5073e06c-cc5d-4139-8e9e-786fabbbad11 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:14.217797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:14.224844 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 515/1014] 10.222.0.85 () {56 vars in 1112 bytes} [Tue Aug 30 04:14:14 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:14.229954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83526a6-35e5-46d6-84e3-ecdb17ff65cd None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:14.278800 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41148d94-3f95-4563-8c28-3da69e49e097 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:14.279090 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41148d94-3f95-4563-8c28-3da69e49e097 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:14.279374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41148d94-3f95-4563-8c28-3da69e49e097 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:14.280841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 516/1015] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:14:14.286489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93bcd20-08ca-4e06-a909-b9b51773193d None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:14.286791 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93bcd20-08ca-4e06-a909-b9b51773193d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:14.287046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93bcd20-08ca-4e06-a909-b9b51773193d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:14.290201 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:14.306574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-d93bcd20-08ca-4e06-a909-b9b51773193d None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:14.307777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d93bcd20-08ca-4e06-a909-b9b51773193d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:14.345464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83526a6-35e5-46d6-84e3-ecdb17ff65cd None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:14.383897 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 517/1016] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:14 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:14:14.385841 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3874854-430a-49b6-a262-ddda5c2daadd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:14.386665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3874854-430a-49b6-a262-ddda5c2daadd None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:14.387099 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3874854-430a-49b6-a262-ddda5c2daadd None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:14.387525 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3874854-430a-49b6-a262-ddda5c2daadd None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:14.388315 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3874854-430a-49b6-a262-ddda5c2daadd None admin] RBAC: Adding query filter params (type=compute) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:14.388766 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3874854-430a-49b6-a262-ddda5c2daadd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:14.390332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3874854-430a-49b6-a262-ddda5c2daadd None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:14.396299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 500/1017] 10.222.0.85 () {56 vars in 1112 bytes} [Tue Aug 30 04:14:14 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:14.402313 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93bcd20-08ca-4e06-a909-b9b51773193d None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:14.511499 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93bcd20-08ca-4e06-a909-b9b51773193d None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:14.519829 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763cfee6-12e0-48ae-9178-f0bfc4863d3e None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:14.520415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763cfee6-12e0-48ae-9178-f0bfc4863d3e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:14.520415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763cfee6-12e0-48ae-9178-f0bfc4863d3e None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:14.521997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 501/1018] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:14.526966 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0289e21c-062a-47d3-bcc6-83dc19b9fe93 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:14.527253 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0289e21c-062a-47d3-bcc6-83dc19b9fe93 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:14.527533 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0289e21c-062a-47d3-bcc6-83dc19b9fe93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:14.546700 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-0289e21c-062a-47d3-bcc6-83dc19b9fe93 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:14.547341 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:14.548190 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0289e21c-062a-47d3-bcc6-83dc19b9fe93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:14.560289 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99af3f2a-67a7-4abb-933e-6a56146292d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:14.561194 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99af3f2a-67a7-4abb-933e-6a56146292d6 None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:14.561479 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99af3f2a-67a7-4abb-933e-6a56146292d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:14.561751 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99af3f2a-67a7-4abb-933e-6a56146292d6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:14.562436 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99af3f2a-67a7-4abb-933e-6a56146292d6 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:14.563841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99af3f2a-67a7-4abb-933e-6a56146292d6 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:14.587454 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 518/1019] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:14:14 2022] POST /identity/v3/endpoints => generated 334 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:14:14.649252 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 502/1020] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:14 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:14:14.661661 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99af3f2a-67a7-4abb-933e-6a56146292d6 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:14.775698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99af3f2a-67a7-4abb-933e-6a56146292d6 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:14.822912 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc09a71-b99a-4534-81bb-dddc97e553cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:14.823559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc09a71-b99a-4534-81bb-dddc97e553cb None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:14.823833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc09a71-b99a-4534-81bb-dddc97e553cb None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:14.824108 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc09a71-b99a-4534-81bb-dddc97e553cb None admin] PATH_INFO: `/v3/services/image` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:14.824703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc09a71-b99a-4534-81bb-dddc97e553cb None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:14.826208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc09a71-b99a-4534-81bb-dddc97e553cb None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:14.835849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-dbc09a71-b99a-4534-81bb-dddc97e553cb None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 04:14:14.835849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:14.835849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:14:14.835849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:14:14.835849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:14:14.835849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:14:14.835849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:14:14.835849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:14.835849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 04:14:14.835849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:14.835849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:14:14.835849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:14.835849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:14.835849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:14.835849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:14.835849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:14.835849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:14.837725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:14.837725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:14.837725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:14.837725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:14.837725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:14.837725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:14.837725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:14:14.837725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:14:14.837725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:14:14.837725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:14:14.837725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:14.837725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:14.837725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:14.837725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:14.837725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:14.837725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:14.839031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:14.839031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:14.839031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:14.839031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:14.839031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:14.839031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:14.839031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:14.839031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:14.839031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:14.839031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:14.839031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:14:14.839031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:14.839031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 04:14:14.839031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:14.839031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 519/1021] 10.222.0.85 () {56 vars in 1105 bytes} [Tue Aug 30 04:14:14 2022] GET /identity/v3/services/image => generated 86 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:14.844656 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0289e21c-062a-47d3-bcc6-83dc19b9fe93 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:14.908095 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:14.924913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc09a71-b99a-4534-81bb-dddc97e553cb None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:14.981927 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:14.982132 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0289e21c-062a-47d3-bcc6-83dc19b9fe93 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:15.022499 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079eefec-e857-4295-9d02-c00957d177bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:15.023456 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079eefec-e857-4295-9d02-c00957d177bd None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:15.023903 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079eefec-e857-4295-9d02-c00957d177bd None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:15.024318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079eefec-e857-4295-9d02-c00957d177bd None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:15.025137 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079eefec-e857-4295-9d02-c00957d177bd None admin] RBAC: Adding query filter params (name=image) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:15.025666 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079eefec-e857-4295-9d02-c00957d177bd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:15.027374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079eefec-e857-4295-9d02-c00957d177bd None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:15.035187 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 503/1022] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:14 2022] GET /identity/v3/services?name=image => generated 122 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:15.041091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079eefec-e857-4295-9d02-c00957d177bd None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:15.046876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc09a71-b99a-4534-81bb-dddc97e553cb None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:15.086255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e255a808-61ce-4242-b9d0-4feec350383d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:15.086894 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e255a808-61ce-4242-b9d0-4feec350383d None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:15.087178 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e255a808-61ce-4242-b9d0-4feec350383d None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:15.087446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e255a808-61ce-4242-b9d0-4feec350383d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:15.088551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e255a808-61ce-4242-b9d0-4feec350383d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:15.090016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e255a808-61ce-4242-b9d0-4feec350383d None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:15.096432 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 520/1023] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:15.097064 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e255a808-61ce-4242-b9d0-4feec350383d None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:15.164350 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079eefec-e857-4295-9d02-c00957d177bd None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:15.176783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:15.208958 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb864a5-a07b-42f4-8e98-eb5805fbfdb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:15.209850 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb864a5-a07b-42f4-8e98-eb5805fbfdb2 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:15.210178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb864a5-a07b-42f4-8e98-eb5805fbfdb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:15.210741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb864a5-a07b-42f4-8e98-eb5805fbfdb2 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:15.211653 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb864a5-a07b-42f4-8e98-eb5805fbfdb2 None admin] RBAC: Adding query filter params (type=image) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:15.212189 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb864a5-a07b-42f4-8e98-eb5805fbfdb2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:15.213540 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e255a808-61ce-4242-b9d0-4feec350383d None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:15.214159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb864a5-a07b-42f4-8e98-eb5805fbfdb2 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:15.221088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 504/1024] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:15 2022] GET /identity/v3/services?type=image => generated 350 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:15.226795 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb864a5-a07b-42f4-8e98-eb5805fbfdb2 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:15.235459 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:15.260622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13516f0f-4748-477f-ad4a-de7fb5cba43e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:15.261540 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13516f0f-4748-477f-ad4a-de7fb5cba43e None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:15.261992 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13516f0f-4748-477f-ad4a-de7fb5cba43e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:15.262300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13516f0f-4748-477f-ad4a-de7fb5cba43e None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:15.262909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13516f0f-4748-477f-ad4a-de7fb5cba43e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:15.264337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13516f0f-4748-477f-ad4a-de7fb5cba43e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:15.293885 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 521/1025] 10.222.0.85 () {60 vars in 1141 bytes} [Tue Aug 30 04:14:15 2022] POST /identity/v3/projects => generated 311 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:14:15.354713 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb864a5-a07b-42f4-8e98-eb5805fbfdb2 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:15.395282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5f6984-f1c1-42d1-98b2-fe3ec6ab3fe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:15.395939 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5f6984-f1c1-42d1-98b2-fe3ec6ab3fe5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:15.396183 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5f6984-f1c1-42d1-98b2-fe3ec6ab3fe5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:15.396507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5f6984-f1c1-42d1-98b2-fe3ec6ab3fe5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:15.397265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5f6984-f1c1-42d1-98b2-fe3ec6ab3fe5 None admin] RBAC: Adding query filter params (interface=public, service_id=aad2a3a176034421915c1f39d7f92425, region_id=RegionOne) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:15.397587 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5f6984-f1c1-42d1-98b2-fe3ec6ab3fe5 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:15.398967 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5f6984-f1c1-42d1-98b2-fe3ec6ab3fe5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:15.405164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 505/1026] 10.222.0.85 () {56 vars in 1251 bytes} [Tue Aug 30 04:14:15 2022] GET /identity/v3/endpoints?service_id=aad2a3a176034421915c1f39d7f92425&interface=public®ion_id=RegionOne => generated 194 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:15.411361 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13516f0f-4748-477f-ad4a-de7fb5cba43e None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:15.531316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13516f0f-4748-477f-ad4a-de7fb5cba43e None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:15.568208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cf8b7c-738d-4842-83f8-5c8cf428eb3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:15.568813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cf8b7c-738d-4842-83f8-5c8cf428eb3e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:15.569063 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cf8b7c-738d-4842-83f8-5c8cf428eb3e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:15.569306 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cf8b7c-738d-4842-83f8-5c8cf428eb3e None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:15.569920 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cf8b7c-738d-4842-83f8-5c8cf428eb3e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:15.571429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cf8b7c-738d-4842-83f8-5c8cf428eb3e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:15.579487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 522/1027] 10.222.0.85 () {56 vars in 1087 bytes} [Tue Aug 30 04:14:15 2022] GET /identity/v3/services => generated 1957 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:14:16.662207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aada3ee1-d5fe-437b-a6ab-7a6022e7c05a None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:16.662733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aada3ee1-d5fe-437b-a6ab-7a6022e7c05a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:16.662733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aada3ee1-d5fe-437b-a6ab-7a6022e7c05a None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:16.664238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 506/1028] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:16.669159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83d3d71-fa62-403f-b941-49eefccc7ccf None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:16.669582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83d3d71-fa62-403f-b941-49eefccc7ccf None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:16.669752 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83d3d71-fa62-403f-b941-49eefccc7ccf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:16.687944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-c83d3d71-fa62-403f-b941-49eefccc7ccf None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:16.689216 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c83d3d71-fa62-403f-b941-49eefccc7ccf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:16.718401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:16.762811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 523/1029] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:16 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:17.025900 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aada3ee1-d5fe-437b-a6ab-7a6022e7c05a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:17.064454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:17.136001 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aada3ee1-d5fe-437b-a6ab-7a6022e7c05a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:17.172435 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea31d89-b049-4a88-b552-37b819196480 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:17.172903 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea31d89-b049-4a88-b552-37b819196480 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:17.173214 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea31d89-b049-4a88-b552-37b819196480 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:17.173492 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea31d89-b049-4a88-b552-37b819196480 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:17.184119 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea31d89-b049-4a88-b552-37b819196480 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:14:17.188977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-6ea31d89-b049-4a88-b552-37b819196480 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 04:14:17.188977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:17.188977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:17.188977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:17.188977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:17.188977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:17.188977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:17.188977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:17.188977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:17.188977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:17.188977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:17.188977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:17.188977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:14:17.188977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:14:17.188977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:14:17.188977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:14:17.189909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:14:17.189909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:14:17.189909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:14:17.189909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:14:17.189909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:17.189909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:17.189909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:17.189909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:17.189909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:17.189909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:17.189909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:17.189909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:17.189909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:17.189909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:17.189909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:17.189909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:17.193925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:17.193925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:17.193925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:17.193925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:17.193925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:14:17.193925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:14:17.193925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:14:17.193925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:14:17.193925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:14:17.193925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:14:17.193925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 04:14:17.193925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:17.193925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 507/1030] 10.222.0.85 () {56 vars in 1120 bytes} [Tue Aug 30 04:14:17 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:17.196081 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83d3d71-fa62-403f-b941-49eefccc7ccf None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:17.269736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595640e1-efdc-4e9f-a976-512e21eafeb0 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:17.270030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595640e1-efdc-4e9f-a976-512e21eafeb0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:17.270284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595640e1-efdc-4e9f-a976-512e21eafeb0 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:17.272446 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 508/1031] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:17.277256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ae2c37-d847-495e-8819-f5751161d2d6 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:17.277479 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ae2c37-d847-495e-8819-f5751161d2d6 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:17.277751 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ae2c37-d847-495e-8819-f5751161d2d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:17.296006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-18ae2c37-d847-495e-8819-f5751161d2d6 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:17.297229 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-18ae2c37-d847-495e-8819-f5751161d2d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:17.314202 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83d3d71-fa62-403f-b941-49eefccc7ccf None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:17.351466 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea4af75-2326-487a-9ea2-3842500422ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:17.352075 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea4af75-2326-487a-9ea2-3842500422ab None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:17.352354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea4af75-2326-487a-9ea2-3842500422ab None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:17.352625 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea4af75-2326-487a-9ea2-3842500422ab None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:17.353320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea4af75-2326-487a-9ea2-3842500422ab None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:17.353619 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea4af75-2326-487a-9ea2-3842500422ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:17.355090 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea4af75-2326-487a-9ea2-3842500422ab None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:17.363793 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 524/1032] 10.222.0.85 () {56 vars in 1115 bytes} [Tue Aug 30 04:14:17 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:17.373067 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea4af75-2326-487a-9ea2-3842500422ab None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:17.373216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 509/1033] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:17 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:14:17.396598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:17.475374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea4af75-2326-487a-9ea2-3842500422ab None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:17.510690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d019f2c7-e165-470f-b594-0cd142ccf6f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:17.511318 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d019f2c7-e165-470f-b594-0cd142ccf6f1 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:17.511595 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d019f2c7-e165-470f-b594-0cd142ccf6f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:17.511870 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d019f2c7-e165-470f-b594-0cd142ccf6f1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:17.522684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d019f2c7-e165-470f-b594-0cd142ccf6f1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:17.524149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d019f2c7-e165-470f-b594-0cd142ccf6f1 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:17.538886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-d019f2c7-e165-470f-b594-0cd142ccf6f1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:17.538886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:17.538886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:17.538886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:17.538886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:17.538886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:17.538886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:17.538886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:17.538886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:17.538886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:17.538886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:17.538886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:17.538886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:14:17.538886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:14:17.538886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:14:17.538886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:14:17.539824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:17.539824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:17.539824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:17.539824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:17.539824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:17.539824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:17.539824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:17.539824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:17.539824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:17.539824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:17.539824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:17.539824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:17.539824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:17.539824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:17.539824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:17.539824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:17.539824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:14:17.545802 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:14:17.545802 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:17.545802 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:17.545802 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 525/1034] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:14:17.546031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ae2c37-d847-495e-8819-f5751161d2d6 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:17.547521 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658eef55-0927-4eea-98c0-063f66785196 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:17.547802 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658eef55-0927-4eea-98c0-063f66785196 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:17.548074 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658eef55-0927-4eea-98c0-063f66785196 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:17.549389 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 526/1035] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:17.554965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464966ca-625c-45a2-b404-4e0d72b15678 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:17.555249 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464966ca-625c-45a2-b404-4e0d72b15678 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:17.555521 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464966ca-625c-45a2-b404-4e0d72b15678 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:17.573485 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-464966ca-625c-45a2-b404-4e0d72b15678 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:17.574675 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-464966ca-625c-45a2-b404-4e0d72b15678 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:17.575105 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:17.647792 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 527/1036] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:17 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:14:17.649241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464966ca-625c-45a2-b404-4e0d72b15678 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:17.658839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:17.664567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ae2c37-d847-495e-8819-f5751161d2d6 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:17.700716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e351f70-d541-4510-9d86-5c46f8f22c3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:17.701349 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e351f70-d541-4510-9d86-5c46f8f22c3c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:17.701630 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e351f70-d541-4510-9d86-5c46f8f22c3c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:17.701911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e351f70-d541-4510-9d86-5c46f8f22c3c None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:17.702552 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e351f70-d541-4510-9d86-5c46f8f22c3c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:17.702851 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e351f70-d541-4510-9d86-5c46f8f22c3c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:17.704223 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e351f70-d541-4510-9d86-5c46f8f22c3c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:17.716800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 510/1037] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:14:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:17.722067 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e351f70-d541-4510-9d86-5c46f8f22c3c None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:17.771455 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464966ca-625c-45a2-b404-4e0d72b15678 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:17.809843 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc5e3cf-96f0-4550-92bb-7687fc2c069e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:17.810390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc5e3cf-96f0-4550-92bb-7687fc2c069e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:17.810667 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc5e3cf-96f0-4550-92bb-7687fc2c069e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:17.810926 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc5e3cf-96f0-4550-92bb-7687fc2c069e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:17.812017 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc5e3cf-96f0-4550-92bb-7687fc2c069e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:17.813468 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc5e3cf-96f0-4550-92bb-7687fc2c069e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:17.815098 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 528/1038] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:17 2022] GET /identity/v3/domains/default => generated 199 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:17.819646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:17.821239 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc5e3cf-96f0-4550-92bb-7687fc2c069e None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:17.838338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e351f70-d541-4510-9d86-5c46f8f22c3c None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:17.875710 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234ac522-ba05-4ebd-a7dd-34db6e7da52e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:17.876349 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234ac522-ba05-4ebd-a7dd-34db6e7da52e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:17.876628 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234ac522-ba05-4ebd-a7dd-34db6e7da52e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:17.876901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234ac522-ba05-4ebd-a7dd-34db6e7da52e None admin] PATH_INFO: `/v3/users/nova` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:17.886148 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234ac522-ba05-4ebd-a7dd-34db6e7da52e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:17.887600 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234ac522-ba05-4ebd-a7dd-34db6e7da52e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:17.899698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-234ac522-ba05-4ebd-a7dd-34db6e7da52e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:14:17.899698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:17.899698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:17.899698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:17.899698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:17.899698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:17.899698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:17.899698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:17.899698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:17.899698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:17.899698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:17.899698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:17.899698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:14:17.899698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:14:17.899698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:14:17.899698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:14:17.900681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:17.900681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:17.900681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:14:17.900681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:14:17.900681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:14:17.900681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:14:17.900681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:17.900681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:17.900681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:17.900681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:17.900681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:17.900681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:17.900681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:17.900681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:17.900681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:17.900681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:17.900681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:17.900681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:17.902128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:17.902128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:17.902128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:14:17.902128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:14:17.902128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:14:17.902128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:14:17.902128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:14:17.902128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:14:17.902128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:14:17.902128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:17.902128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 511/1039] 10.222.0.85 () {56 vars in 1093 bytes} [Tue Aug 30 04:14:17 2022] GET /identity/v3/users/nova => generated 82 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:17.903937 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:17.906532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234ac522-ba05-4ebd-a7dd-34db6e7da52e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:17.931291 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc5e3cf-96f0-4550-92bb-7687fc2c069e None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:17.965366 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4996766d-cc13-47ad-a671-6e41fc385310 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:17.966117 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4996766d-cc13-47ad-a671-6e41fc385310 None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:17.966504 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4996766d-cc13-47ad-a671-6e41fc385310 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:17.966787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4996766d-cc13-47ad-a671-6e41fc385310 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:17.967434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4996766d-cc13-47ad-a671-6e41fc385310 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:17.968864 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4996766d-cc13-47ad-a671-6e41fc385310 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:17.992921 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 529/1040] 10.222.0.85 () {60 vars in 1132 bytes} [Tue Aug 30 04:14:17 2022] POST /identity/v3/users => generated 285 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:14:17.993806 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4996766d-cc13-47ad-a671-6e41fc385310 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:18.017972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234ac522-ba05-4ebd-a7dd-34db6e7da52e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:18.058846 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7652050-beda-4703-a333-696509235fab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:18.059476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7652050-beda-4703-a333-696509235fab None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:18.059750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7652050-beda-4703-a333-696509235fab None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:18.060081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7652050-beda-4703-a333-696509235fab None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:18.060771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7652050-beda-4703-a333-696509235fab None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:18.061241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7652050-beda-4703-a333-696509235fab None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:18.062882 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7652050-beda-4703-a333-696509235fab None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:18.089404 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 512/1041] 10.222.0.85 () {56 vars in 1133 bytes} [Tue Aug 30 04:14:17 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:18.090623 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7652050-beda-4703-a333-696509235fab None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:18.094566 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:18.123217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4996766d-cc13-47ad-a671-6e41fc385310 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:18.154846 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:18.164812 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c212a303-28f1-41ca-8b77-ebcd3895084d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:18.165490 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c212a303-28f1-41ca-8b77-ebcd3895084d None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:18.165768 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c212a303-28f1-41ca-8b77-ebcd3895084d None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:18.166041 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c212a303-28f1-41ca-8b77-ebcd3895084d None admin] PATH_INFO: `/v3/services/image` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:18.166633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c212a303-28f1-41ca-8b77-ebcd3895084d None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:18.167996 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c212a303-28f1-41ca-8b77-ebcd3895084d None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:18.178377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-c212a303-28f1-41ca-8b77-ebcd3895084d None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 04:14:18.178377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:18.178377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:14:18.178377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:14:18.178377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:14:18.178377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:14:18.178377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:14:18.178377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:18.178377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 04:14:18.178377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:18.178377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:14:18.178377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:18.178377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:18.178377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:18.178377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:18.178377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:18.178377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:18.179393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:18.179393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:18.179393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:18.179393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:18.179393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:18.179393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:18.179393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:14:18.179393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:14:18.179393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:14:18.179393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:14:18.179393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:18.179393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:18.179393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:18.179393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:18.179393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:18.179393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:18.180266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:18.180266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:18.180266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:18.180266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:18.180266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:18.180266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:18.180266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:18.180266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:18.180266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:18.180266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:18.180266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:14:18.180266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:14:18.180266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 04:14:18.180266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:18.180266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 530/1042] 10.222.0.85 () {56 vars in 1105 bytes} [Tue Aug 30 04:14:17 2022] GET /identity/v3/services/image => generated 86 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:18.186593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c212a303-28f1-41ca-8b77-ebcd3895084d None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:18.225893 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7652050-beda-4703-a333-696509235fab None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:18.263296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31cb89b-dd61-496b-bbb3-b83c3e3781ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:18.264203 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31cb89b-dd61-496b-bbb3-b83c3e3781ac None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:18.264660 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31cb89b-dd61-496b-bbb3-b83c3e3781ac None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:18.265181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31cb89b-dd61-496b-bbb3-b83c3e3781ac None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:18.303104 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c212a303-28f1-41ca-8b77-ebcd3895084d None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:18.336310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31cb89b-dd61-496b-bbb3-b83c3e3781ac None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:18.337282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd9c599-8958-4a53-8b92-b86d153110cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:18.337697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31cb89b-dd61-496b-bbb3-b83c3e3781ac None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:18.337884 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd9c599-8958-4a53-8b92-b86d153110cc None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:18.338166 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd9c599-8958-4a53-8b92-b86d153110cc None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:18.338437 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd9c599-8958-4a53-8b92-b86d153110cc None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:18.339070 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd9c599-8958-4a53-8b92-b86d153110cc None admin] RBAC: Adding query filter params (name=image) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:18.339371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd9c599-8958-4a53-8b92-b86d153110cc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:18.340546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:18.340723 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd9c599-8958-4a53-8b92-b86d153110cc None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:18.345959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 531/1043] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:18 2022] GET /identity/v3/services?name=image => generated 122 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:18.352279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd9c599-8958-4a53-8b92-b86d153110cc None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:18.416226 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:18.448787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 513/1044] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:14:18 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 04:14:18.454320 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31cb89b-dd61-496b-bbb3-b83c3e3781ac None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:18.466407 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd9c599-8958-4a53-8b92-b86d153110cc None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:18.506328 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1253fa-6efb-4cc8-9884-13027f0cb4d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:18.507442 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1253fa-6efb-4cc8-9884-13027f0cb4d0 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:18.507717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1253fa-6efb-4cc8-9884-13027f0cb4d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:18.507988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1253fa-6efb-4cc8-9884-13027f0cb4d0 None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:18.508617 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1253fa-6efb-4cc8-9884-13027f0cb4d0 None admin] RBAC: Adding query filter params (type=image) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:18.508912 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1253fa-6efb-4cc8-9884-13027f0cb4d0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:18.510810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1253fa-6efb-4cc8-9884-13027f0cb4d0 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:18.521219 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 532/1045] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:18 2022] GET /identity/v3/services?type=image => generated 350 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:18.522240 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1253fa-6efb-4cc8-9884-13027f0cb4d0 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:18.561060 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31cb89b-dd61-496b-bbb3-b83c3e3781ac None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:18.564122 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:18.598249 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacb70b8-aab7-493a-8ebd-0081d82cb2ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:18.598796 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacb70b8-aab7-493a-8ebd-0081d82cb2ad None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:18.599074 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacb70b8-aab7-493a-8ebd-0081d82cb2ad None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:18.599341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacb70b8-aab7-493a-8ebd-0081d82cb2ad None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:18.609310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacb70b8-aab7-493a-8ebd-0081d82cb2ad None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:18.610750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacb70b8-aab7-493a-8ebd-0081d82cb2ad None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:18.624888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-bacb70b8-aab7-493a-8ebd-0081d82cb2ad None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:18.624888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:18.624888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:18.624888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:18.624888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:18.624888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:18.624888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:18.624888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:18.624888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:18.624888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:18.624888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:18.624888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:18.624888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:14:18.624888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:14:18.624888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:14:18.624888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:14:18.625886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:18.625886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:18.625886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:18.625886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:18.625886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:18.625886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:18.625886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:18.625886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:18.625886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:18.625886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:18.625886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:18.625886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:18.625886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:18.625886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:18.625886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:18.625886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:18.625886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:14:18.627586 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:14:18.627586 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:18.627586 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:18.627586 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 514/1046] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:18.632565 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacb70b8-aab7-493a-8ebd-0081d82cb2ad None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:18.639234 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1253fa-6efb-4cc8-9884-13027f0cb4d0 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:18.664224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:18.677970 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a436c48-4d32-44c5-a696-dd5e8dada011 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:18.678681 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a436c48-4d32-44c5-a696-dd5e8dada011 None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:18.678959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a436c48-4d32-44c5-a696-dd5e8dada011 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:18.679231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a436c48-4d32-44c5-a696-dd5e8dada011 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:18.679894 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a436c48-4d32-44c5-a696-dd5e8dada011 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:18.681286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a436c48-4d32-44c5-a696-dd5e8dada011 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:18.703210 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 533/1047] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:14:18 2022] POST /identity/v3/endpoints => generated 327 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:14:18.765583 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacb70b8-aab7-493a-8ebd-0081d82cb2ad None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:18.810355 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f1ab0d-219e-4820-95c1-56d53bd3913f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:18.810979 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f1ab0d-219e-4820-95c1-56d53bd3913f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:18.811260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f1ab0d-219e-4820-95c1-56d53bd3913f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:18.811533 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f1ab0d-219e-4820-95c1-56d53bd3913f None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:18.812196 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f1ab0d-219e-4820-95c1-56d53bd3913f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:18.812499 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f1ab0d-219e-4820-95c1-56d53bd3913f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:18.813932 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f1ab0d-219e-4820-95c1-56d53bd3913f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:18.831010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 515/1048] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:14:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:18.837473 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a436c48-4d32-44c5-a696-dd5e8dada011 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:18.945100 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a436c48-4d32-44c5-a696-dd5e8dada011 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:18.981098 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce67f647-aea7-4923-95b6-170aca9f65e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:18.981735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce67f647-aea7-4923-95b6-170aca9f65e6 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:18.982012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce67f647-aea7-4923-95b6-170aca9f65e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:18.982311 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce67f647-aea7-4923-95b6-170aca9f65e6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:18.988968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce67f647-aea7-4923-95b6-170aca9f65e6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:18.990440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce67f647-aea7-4923-95b6-170aca9f65e6 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:19.001722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-ce67f647-aea7-4923-95b6-170aca9f65e6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:14:19.001722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:19.001722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:19.001722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:19.001722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:19.001722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:19.001722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:19.001722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:19.001722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:19.001722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:19.001722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:19.001722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:19.001722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:14:19.001722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:14:19.001722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:14:19.002790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:14:19.002790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:19.002790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:19.002790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:19.002790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:19.002790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:19.002790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:19.002790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:19.002790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:19.002790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:19.002790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:19.002790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:19.002790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:19.002790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:19.002790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:19.002790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:19.002790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:19.004377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:14:19.004377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:14:19.004377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:14:19.004377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:14:19.004377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:14:19.004377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:14:19.004377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:14:19.004377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:19.004377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 534/1049] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:14:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:14:19.010979 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f1ab0d-219e-4820-95c1-56d53bd3913f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:19.026989 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:19.121176 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f1ab0d-219e-4820-95c1-56d53bd3913f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:19.158564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddaa1467-77d4-407b-943e-6eb642f54020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:19.159195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaa1467-77d4-407b-943e-6eb642f54020 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:19.159476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaa1467-77d4-407b-943e-6eb642f54020 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:19.160248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaa1467-77d4-407b-943e-6eb642f54020 None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:19.160953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddaa1467-77d4-407b-943e-6eb642f54020 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:19.161295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddaa1467-77d4-407b-943e-6eb642f54020 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:19.162886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddaa1467-77d4-407b-943e-6eb642f54020 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:19.175195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 516/1050] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:14:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:19.181137 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce67f647-aea7-4923-95b6-170aca9f65e6 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:19.223079 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:19.292909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce67f647-aea7-4923-95b6-170aca9f65e6 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:19.326250 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847f4409-c940-4eb9-aa6f-d77c2151ff1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:19.326846 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847f4409-c940-4eb9-aa6f-d77c2151ff1a None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:19.327120 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847f4409-c940-4eb9-aa6f-d77c2151ff1a None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:19.327387 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847f4409-c940-4eb9-aa6f-d77c2151ff1a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:19.328100 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847f4409-c940-4eb9-aa6f-d77c2151ff1a None admin] RBAC: Adding query filter params (role.id=6b76b888c73142da957a10d32ce5e995, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=4bc4d455ebf041d488b1199ca58358a4) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:19.328400 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847f4409-c940-4eb9-aa6f-d77c2151ff1a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:19.331422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847f4409-c940-4eb9-aa6f-d77c2151ff1a None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:19.345314 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 535/1051] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:14:19 2022] GET /identity/v3/role_assignments?user.id=4bc4d455ebf041d488b1199ca58358a4&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=6b76b888c73142da957a10d32ce5e995 => generated 259 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:19.982421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f1eaaa-06e2-416f-bafd-983544848f3d None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:19.982943 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f1eaaa-06e2-416f-bafd-983544848f3d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:19.983050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f1eaaa-06e2-416f-bafd-983544848f3d None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:19.984358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 517/1052] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:19.989724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fffaade-25a0-4188-8f49-a4c23899e519 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:19.990152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fffaade-25a0-4188-8f49-a4c23899e519 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:19.990296 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fffaade-25a0-4188-8f49-a4c23899e519 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:20.009976 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-7fffaade-25a0-4188-8f49-a4c23899e519 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:20.011163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-7fffaade-25a0-4188-8f49-a4c23899e519 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:20.089933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 536/1053] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:20.359942 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f1eaaa-06e2-416f-bafd-983544848f3d None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:20.444497 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:20.477141 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f1eaaa-06e2-416f-bafd-983544848f3d None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:20.512042 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71d7864-b97b-4e25-a399-72300e374371 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:20.512677 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71d7864-b97b-4e25-a399-72300e374371 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:20.513084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71d7864-b97b-4e25-a399-72300e374371 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:20.513371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71d7864-b97b-4e25-a399-72300e374371 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:20.524726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71d7864-b97b-4e25-a399-72300e374371 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:14:20.529581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-e71d7864-b97b-4e25-a399-72300e374371 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:14:20.529581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:20.529581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:20.529581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:20.529581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:20.529581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:20.529581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:20.529581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:20.529581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:20.529581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:20.529581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:20.529581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:20.529581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:14:20.529581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:14:20.529581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:14:20.529581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:14:20.530478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:14:20.530478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:14:20.530478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:14:20.530478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:14:20.530478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:20.530478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:20.530478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:20.530478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:20.530478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:20.530478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:20.530478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:20.530478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:20.530478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:20.530478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:20.530478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:20.530478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:20.531348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:20.531348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:20.531348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:20.531348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:20.531348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:14:20.531348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:14:20.531348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:14:20.531348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:14:20.531348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:14:20.531348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:14:20.531348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:14:20.531348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:20.533430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 518/1054] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:14:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:20.537237 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fffaade-25a0-4188-8f49-a4c23899e519 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:20.567919 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:20.632777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6758e85b-f9c1-4176-b8be-422437167b9c None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:20.632957 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6758e85b-f9c1-4176-b8be-422437167b9c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:20.633288 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6758e85b-f9c1-4176-b8be-422437167b9c None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:20.634744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 519/1055] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:14:20.638785 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fffaade-25a0-4188-8f49-a4c23899e519 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:20.639248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357ae5dc-473c-49e6-870e-bce3164a184f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:20.639527 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357ae5dc-473c-49e6-870e-bce3164a184f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:20.639793 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357ae5dc-473c-49e6-870e-bce3164a184f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:20.657857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-357ae5dc-473c-49e6-870e-bce3164a184f None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:20.659033 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-357ae5dc-473c-49e6-870e-bce3164a184f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:20.677153 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462c92a7-4bc5-4534-8c8a-89cc12498297 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:20.677799 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462c92a7-4bc5-4534-8c8a-89cc12498297 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:20.678325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462c92a7-4bc5-4534-8c8a-89cc12498297 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:20.678593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462c92a7-4bc5-4534-8c8a-89cc12498297 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:20.679257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462c92a7-4bc5-4534-8c8a-89cc12498297 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:20.679555 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462c92a7-4bc5-4534-8c8a-89cc12498297 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:20.681367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462c92a7-4bc5-4534-8c8a-89cc12498297 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:20.696781 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 537/1056] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:14:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:20.698233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462c92a7-4bc5-4534-8c8a-89cc12498297 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:20.729212 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 520/1057] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:14:20.797756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:20.806377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462c92a7-4bc5-4534-8c8a-89cc12498297 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:20.844959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0e91c7-c9e0-49c1-a06c-51c4b9be51c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:20.845635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0e91c7-c9e0-49c1-a06c-51c4b9be51c8 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:20.845912 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0e91c7-c9e0-49c1-a06c-51c4b9be51c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:20.846182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0e91c7-c9e0-49c1-a06c-51c4b9be51c8 None admin] PATH_INFO: `/v3/users/8e5773379fb44ac98c91dd141dcdbeac` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:20.860407 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0e91c7-c9e0-49c1-a06c-51c4b9be51c8 None admin] RBAC: Authorizing `identity:get_user(user_id=8e5773379fb44ac98c91dd141dcdbeac)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:20.861897 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0e91c7-c9e0-49c1-a06c-51c4b9be51c8 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:20.868840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 538/1058] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:14:20 2022] GET /identity/v3/users/8e5773379fb44ac98c91dd141dcdbeac => generated 285 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:20.870011 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357ae5dc-473c-49e6-870e-bce3164a184f None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:20.934847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:20.976475 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357ae5dc-473c-49e6-870e-bce3164a184f None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:21.012892 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e576787-313d-4143-be90-18125c82f3ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:21.013578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e576787-313d-4143-be90-18125c82f3ab None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:21.013815 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e576787-313d-4143-be90-18125c82f3ab None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:21.014076 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e576787-313d-4143-be90-18125c82f3ab None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:21.015873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0e91c7-c9e0-49c1-a06c-51c4b9be51c8 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:21.086288 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e576787-313d-4143-be90-18125c82f3ab None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:21.087746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e576787-313d-4143-be90-18125c82f3ab None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:21.130046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0e91c7-c9e0-49c1-a06c-51c4b9be51c8 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:21.154736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:21.168445 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cacdfa-10c6-4a6a-a76f-269da4d3f5c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:21.169073 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cacdfa-10c6-4a6a-a76f-269da4d3f5c3 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:21.169349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cacdfa-10c6-4a6a-a76f-269da4d3f5c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:21.169613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cacdfa-10c6-4a6a-a76f-269da4d3f5c3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:21.193743 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:21.204350 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 521/1059] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:14:20 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 04:14:21.207451 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e576787-313d-4143-be90-18125c82f3ab None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:21.241685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cacdfa-10c6-4a6a-a76f-269da4d3f5c3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:21.243152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cacdfa-10c6-4a6a-a76f-269da4d3f5c3 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:21.317783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e576787-313d-4143-be90-18125c82f3ab None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:21.342787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 539/1060] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:14:21 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 04:14:21.344291 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb83431-8276-49a2-ac42-b3444d43cc0e None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:21.344582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb83431-8276-49a2-ac42-b3444d43cc0e None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:21.344857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb83431-8276-49a2-ac42-b3444d43cc0e None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:21.346131 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 540/1061] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:21.348547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb83431-8276-49a2-ac42-b3444d43cc0e None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:21.351885 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a46191-8fc3-413b-b34e-08bbc76c9dd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:21.352375 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a46191-8fc3-413b-b34e-08bbc76c9dd6 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:21.352654 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a46191-8fc3-413b-b34e-08bbc76c9dd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:21.352927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a46191-8fc3-413b-b34e-08bbc76c9dd6 None admin] PATH_INFO: `/v3/projects/f9ffd2c237014adcabcbab50d24d6134` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:21.354105 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a46191-8fc3-413b-b34e-08bbc76c9dd6 None admin] RBAC: Authorizing `identity:get_project(project_id=f9ffd2c237014adcabcbab50d24d6134)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:21.355537 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a46191-8fc3-413b-b34e-08bbc76c9dd6 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:21.357306 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 522/1062] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:14:21 2022] GET /identity/v3/projects/f9ffd2c237014adcabcbab50d24d6134 => generated 311 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:21.358294 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb85df5-b374-45be-a6a0-6b33b94d1bd1 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:21.358577 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb85df5-b374-45be-a6a0-6b33b94d1bd1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:21.358852 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb85df5-b374-45be-a6a0-6b33b94d1bd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:21.376247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-0bb85df5-b374-45be-a6a0-6b33b94d1bd1 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:21.377476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0bb85df5-b374-45be-a6a0-6b33b94d1bd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:21.402245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:21.405070 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:21.445805 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 523/1063] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:21.446564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb85df5-b374-45be-a6a0-6b33b94d1bd1 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:21.457323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb83431-8276-49a2-ac42-b3444d43cc0e None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:21.494311 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad80706c-dd2e-4639-a0d3-bcd6aeff7512 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:21.495050 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad80706c-dd2e-4639-a0d3-bcd6aeff7512 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:21.495397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad80706c-dd2e-4639-a0d3-bcd6aeff7512 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:21.495739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad80706c-dd2e-4639-a0d3-bcd6aeff7512 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:21.507949 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad80706c-dd2e-4639-a0d3-bcd6aeff7512 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:21.509481 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad80706c-dd2e-4639-a0d3-bcd6aeff7512 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:21.524804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-ad80706c-dd2e-4639-a0d3-bcd6aeff7512 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:21.524804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:21.524804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:21.524804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:21.524804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:21.524804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:21.524804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:21.524804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:21.524804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:21.524804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:21.524804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:21.524804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:21.524804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:14:21.524804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:14:21.524804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:14:21.524804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:14:21.525847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:21.525847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:21.525847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:21.525847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:21.525847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:21.525847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:21.525847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:21.525847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:21.525847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:21.525847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:21.525847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:21.525847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:21.525847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:21.525847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:21.525847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:21.525847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:21.525847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:14:21.527582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:14:21.527582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:21.527582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:21.527582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 541/1064] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:14:21.531789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad80706c-dd2e-4639-a0d3-bcd6aeff7512 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:21.564785 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb85df5-b374-45be-a6a0-6b33b94d1bd1 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:21.600475 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61ceadb-0d93-42d2-be78-39734a225b6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:21.601104 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61ceadb-0d93-42d2-be78-39734a225b6b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:21.601390 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61ceadb-0d93-42d2-be78-39734a225b6b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:21.601666 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61ceadb-0d93-42d2-be78-39734a225b6b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:21.602399 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61ceadb-0d93-42d2-be78-39734a225b6b None admin] RBAC: Adding query filter params (role.id=d17954e7ce544f0d90a512340232c60e, scope.project.id=f9ffd2c237014adcabcbab50d24d6134, user.id=8e5773379fb44ac98c91dd141dcdbeac) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:21.602708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61ceadb-0d93-42d2-be78-39734a225b6b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:21.604144 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61ceadb-0d93-42d2-be78-39734a225b6b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:21.609599 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 524/1065] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:14:21 2022] GET /identity/v3/role_assignments?user.id=8e5773379fb44ac98c91dd141dcdbeac&scope.project.id=f9ffd2c237014adcabcbab50d24d6134&role.id=d17954e7ce544f0d90a512340232c60e => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:21.639316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad80706c-dd2e-4639-a0d3-bcd6aeff7512 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:21.642023 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:21.680066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf26020-cef0-493a-ba24-c030ce2f6ff6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:21.680687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf26020-cef0-493a-ba24-c030ce2f6ff6 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:21.680966 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf26020-cef0-493a-ba24-c030ce2f6ff6 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:21.681282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf26020-cef0-493a-ba24-c030ce2f6ff6 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:21.681944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf26020-cef0-493a-ba24-c030ce2f6ff6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:21.682246 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf26020-cef0-493a-ba24-c030ce2f6ff6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:21.683642 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf26020-cef0-493a-ba24-c030ce2f6ff6 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:21.709159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 542/1066] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:14:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:21.714113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61ceadb-0d93-42d2-be78-39734a225b6b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:21.751935 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:21.842513 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61ceadb-0d93-42d2-be78-39734a225b6b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:21.879926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15180b6e-2ac1-495c-b0d5-aeb789fa32f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:21.880567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15180b6e-2ac1-495c-b0d5-aeb789fa32f3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:21.881128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15180b6e-2ac1-495c-b0d5-aeb789fa32f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:21.881405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15180b6e-2ac1-495c-b0d5-aeb789fa32f3 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:21.892362 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15180b6e-2ac1-495c-b0d5-aeb789fa32f3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:14:21.897303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-15180b6e-2ac1-495c-b0d5-aeb789fa32f3 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 04:14:21.897303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:21.897303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:21.897303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:21.897303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:21.897303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:21.897303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:21.897303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:21.897303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:21.897303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:21.897303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:21.897303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:21.897303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:14:21.897303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:14:21.897303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:14:21.897303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:14:21.898238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:14:21.898238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:14:21.898238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:14:21.898238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:14:21.898238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:21.898238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:21.898238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:21.898238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:21.898238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:21.898238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:21.898238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:21.898238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:21.898238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:21.898238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:21.898238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:21.898238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:21.899085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:21.899085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:21.899085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:21.899085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:21.899085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:14:21.899085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:14:21.899085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:14:21.899085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:14:21.899085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:14:21.899085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:14:21.899085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 04:14:21.899085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:21.899085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 525/1067] 10.222.0.85 () {56 vars in 1120 bytes} [Tue Aug 30 04:14:21 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:21.904351 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf26020-cef0-493a-ba24-c030ce2f6ff6 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:22.020507 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf26020-cef0-493a-ba24-c030ce2f6ff6 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:22.055298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3079363a-6689-4d2d-bef5-7b03aa5df03a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:22.055902 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3079363a-6689-4d2d-bef5-7b03aa5df03a None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:22.056175 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3079363a-6689-4d2d-bef5-7b03aa5df03a None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:22.056438 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3079363a-6689-4d2d-bef5-7b03aa5df03a None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:22.057116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3079363a-6689-4d2d-bef5-7b03aa5df03a None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:22.057413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3079363a-6689-4d2d-bef5-7b03aa5df03a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:22.058764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3079363a-6689-4d2d-bef5-7b03aa5df03a None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:22.067220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 543/1068] 10.222.0.85 () {56 vars in 1115 bytes} [Tue Aug 30 04:14:21 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:22.073144 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15180b6e-2ac1-495c-b0d5-aeb789fa32f3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:22.164566 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:22.180806 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15180b6e-2ac1-495c-b0d5-aeb789fa32f3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:22.212684 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e4cf55-ee38-4c78-bb9c-3cfc3ee16c0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:22.213338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e4cf55-ee38-4c78-bb9c-3cfc3ee16c0e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:22.213615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e4cf55-ee38-4c78-bb9c-3cfc3ee16c0e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:22.213886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e4cf55-ee38-4c78-bb9c-3cfc3ee16c0e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:22.225007 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e4cf55-ee38-4c78-bb9c-3cfc3ee16c0e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:22.226465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e4cf55-ee38-4c78-bb9c-3cfc3ee16c0e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:22.240639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-d6e4cf55-ee38-4c78-bb9c-3cfc3ee16c0e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:22.240639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:22.240639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:22.240639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:22.240639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:22.240639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:22.240639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:22.240639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:22.240639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:22.240639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:22.240639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:22.240639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:22.240639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:14:22.240639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:14:22.240639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:14:22.240639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:14:22.241639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:22.241639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:22.241639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:22.241639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:22.241639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:22.241639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:22.241639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:22.241639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:22.241639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:22.241639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:22.241639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:22.241639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:22.241639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:22.241639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:22.241639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:22.241639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:22.241639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:14:22.242718 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:14:22.242718 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:22.242718 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:22.242718 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 526/1069] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:22.247627 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3079363a-6689-4d2d-bef5-7b03aa5df03a None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:22.274821 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:22.344986 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3079363a-6689-4d2d-bef5-7b03aa5df03a None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:22.377804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a7212a-5f58-445d-a81c-9b0c02c65722 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:22.378908 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a7212a-5f58-445d-a81c-9b0c02c65722 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:22.379194 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a7212a-5f58-445d-a81c-9b0c02c65722 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:22.379461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a7212a-5f58-445d-a81c-9b0c02c65722 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:22.380090 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a7212a-5f58-445d-a81c-9b0c02c65722 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:22.380386 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a7212a-5f58-445d-a81c-9b0c02c65722 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:22.382020 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a7212a-5f58-445d-a81c-9b0c02c65722 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:22.393307 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 544/1070] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:14:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:22.399494 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e4cf55-ee38-4c78-bb9c-3cfc3ee16c0e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:22.497725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e4cf55-ee38-4c78-bb9c-3cfc3ee16c0e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:22.530925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd71e1c-0710-49d8-8e5f-3baa5b75a9be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:22.531483 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd71e1c-0710-49d8-8e5f-3baa5b75a9be None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:22.531765 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd71e1c-0710-49d8-8e5f-3baa5b75a9be None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:22.532041 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd71e1c-0710-49d8-8e5f-3baa5b75a9be None admin] PATH_INFO: `/v3/users/nova` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:22.534749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:22.540073 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd71e1c-0710-49d8-8e5f-3baa5b75a9be None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:22.541683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd71e1c-0710-49d8-8e5f-3baa5b75a9be None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:22.553524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-fdd71e1c-0710-49d8-8e5f-3baa5b75a9be None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:14:22.553524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:22.553524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:22.553524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:22.553524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:22.553524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:22.553524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:22.553524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:22.553524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:22.553524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:22.553524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:22.553524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:22.553524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:14:22.553524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:14:22.553524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:14:22.553524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:14:22.554617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:22.554617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:22.554617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:14:22.554617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:14:22.554617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:14:22.554617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:14:22.554617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:22.554617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:22.554617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:22.554617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:22.554617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:22.554617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:22.554617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:22.554617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:22.554617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:22.554617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:22.554617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:22.554617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:22.556031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:22.556031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:22.556031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:14:22.556031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:14:22.556031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:14:22.556031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:14:22.556031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:14:22.556031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:14:22.556031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:14:22.556031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:22.556031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 527/1071] 10.222.0.85 () {56 vars in 1093 bytes} [Tue Aug 30 04:14:22 2022] GET /identity/v3/users/nova => generated 82 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:22.560349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a7212a-5f58-445d-a81c-9b0c02c65722 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:22.644304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:22.659338 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a7212a-5f58-445d-a81c-9b0c02c65722 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:22.695532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd05a3ce-0aab-4410-ba04-f16a0a9a9de9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:22.696135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd05a3ce-0aab-4410-ba04-f16a0a9a9de9 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:22.696413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd05a3ce-0aab-4410-ba04-f16a0a9a9de9 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:22.696682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd05a3ce-0aab-4410-ba04-f16a0a9a9de9 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:22.697440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd05a3ce-0aab-4410-ba04-f16a0a9a9de9 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:22.697742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd05a3ce-0aab-4410-ba04-f16a0a9a9de9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:22.699169 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd05a3ce-0aab-4410-ba04-f16a0a9a9de9 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:22.715913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 545/1072] 10.222.0.85 () {56 vars in 1133 bytes} [Tue Aug 30 04:14:22 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:22.721524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd71e1c-0710-49d8-8e5f-3baa5b75a9be None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:22.822796 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd71e1c-0710-49d8-8e5f-3baa5b75a9be None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:22.856099 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242b827d-472f-423c-beeb-199665dec076 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:22.856730 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242b827d-472f-423c-beeb-199665dec076 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:22.857010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242b827d-472f-423c-beeb-199665dec076 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:22.857329 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242b827d-472f-423c-beeb-199665dec076 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:22.868681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242b827d-472f-423c-beeb-199665dec076 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:22.870154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242b827d-472f-423c-beeb-199665dec076 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:22.884317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-242b827d-472f-423c-beeb-199665dec076 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:22.884317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:22.884317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:22.884317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:22.884317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:22.884317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:22.884317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:22.884317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:22.884317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:22.884317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:22.884317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:22.884317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:22.884317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:14:22.884317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:14:22.884317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:14:22.884317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:14:22.885365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:22.885365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:22.885365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:22.885365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:22.885365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:22.885365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:22.885365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:22.885365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:22.885365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:22.885365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:22.885365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:22.885365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:22.885365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:22.885365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:22.885365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:22.885365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:22.885365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:14:22.886278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:14:22.886278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:22.886278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:22.886278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 528/1073] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:22.891410 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd05a3ce-0aab-4410-ba04-f16a0a9a9de9 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:22.993059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd05a3ce-0aab-4410-ba04-f16a0a9a9de9 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:23.026147 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13ffab8-c646-441a-92c9-80e9de80d97f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:23.026756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13ffab8-c646-441a-92c9-80e9de80d97f None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:23.027018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13ffab8-c646-441a-92c9-80e9de80d97f None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:23.027292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13ffab8-c646-441a-92c9-80e9de80d97f None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:23.027929 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13ffab8-c646-441a-92c9-80e9de80d97f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:23.028233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13ffab8-c646-441a-92c9-80e9de80d97f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:23.029642 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13ffab8-c646-441a-92c9-80e9de80d97f None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:23.041362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 546/1074] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:14:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:23.046920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242b827d-472f-423c-beeb-199665dec076 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:23.096819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:23.148841 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242b827d-472f-423c-beeb-199665dec076 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:23.181784 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72b3cc1-db00-4da2-92cd-0dff11cd954f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:23.182437 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72b3cc1-db00-4da2-92cd-0dff11cd954f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:23.182715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72b3cc1-db00-4da2-92cd-0dff11cd954f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:23.182982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72b3cc1-db00-4da2-92cd-0dff11cd954f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:23.189523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72b3cc1-db00-4da2-92cd-0dff11cd954f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:23.190881 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72b3cc1-db00-4da2-92cd-0dff11cd954f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:23.202883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-d72b3cc1-db00-4da2-92cd-0dff11cd954f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:14:23.202883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:23.202883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:23.202883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:23.202883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:23.202883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:23.202883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:23.202883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:23.202883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:23.202883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:23.202883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:23.202883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:23.202883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:14:23.202883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:14:23.202883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:14:23.203921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:14:23.203921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:23.203921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:23.203921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:23.203921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:23.203921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:23.203921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:23.203921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:23.203921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:23.203921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:23.203921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:23.203921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:23.203921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:23.203921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:23.203921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:23.203921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:23.203921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:23.205299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:14:23.205299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:14:23.205299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:14:23.205299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:14:23.205299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:14:23.205299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:14:23.205299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:14:23.205299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:23.205299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 529/1075] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:14:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:23.209872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13ffab8-c646-441a-92c9-80e9de80d97f None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:23.213691 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:23.308892 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13ffab8-c646-441a-92c9-80e9de80d97f None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:23.341655 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e72b28-004e-4d10-ac3d-9537dda6ddd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:23.342246 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e72b28-004e-4d10-ac3d-9537dda6ddd8 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:23.342519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e72b28-004e-4d10-ac3d-9537dda6ddd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:23.342785 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e72b28-004e-4d10-ac3d-9537dda6ddd8 None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:23.343463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e72b28-004e-4d10-ac3d-9537dda6ddd8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:23.343761 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e72b28-004e-4d10-ac3d-9537dda6ddd8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:23.345188 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e72b28-004e-4d10-ac3d-9537dda6ddd8 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:23.361781 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 547/1076] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:14:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:23.363370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72b3cc1-db00-4da2-92cd-0dff11cd954f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:23.464268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72b3cc1-db00-4da2-92cd-0dff11cd954f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:23.473635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:23.498880 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460bc51f-8cf2-4bf6-a3e2-d93e2a2a4c88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:23.499544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460bc51f-8cf2-4bf6-a3e2-d93e2a2a4c88 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:23.499826 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460bc51f-8cf2-4bf6-a3e2-d93e2a2a4c88 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:23.500101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460bc51f-8cf2-4bf6-a3e2-d93e2a2a4c88 None admin] PATH_INFO: `/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/4bc4d455ebf041d488b1199ca58358a4/roles/6b76b888c73142da957a10d32ce5e995` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:23.502207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460bc51f-8cf2-4bf6-a3e2-d93e2a2a4c88 None admin] RBAC: Authorizing `identity:create_grant(project_id=a860dcc3b2d64d1491c40710ec7423a5, user_id=4bc4d455ebf041d488b1199ca58358a4, role_id=6b76b888c73142da957a10d32ce5e995)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:23.504342 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460bc51f-8cf2-4bf6-a3e2-d93e2a2a4c88 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:23.515040 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f8d1b4-8200-456f-b577-5fc207a78d39 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:23.515402 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f8d1b4-8200-456f-b577-5fc207a78d39 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:23.515750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f8d1b4-8200-456f-b577-5fc207a78d39 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:23.517178 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 548/1077] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:23.518992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 530/1078] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:14:23 2022] PUT /identity/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/4bc4d455ebf041d488b1199ca58358a4/roles/6b76b888c73142da957a10d32ce5e995 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:14:23.522354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a48a73-52cd-41c5-b987-bfd7fc74224b None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:23.522706 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a48a73-52cd-41c5-b987-bfd7fc74224b None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:23.523067 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a48a73-52cd-41c5-b987-bfd7fc74224b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:23.544432 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-36a48a73-52cd-41c5-b987-bfd7fc74224b None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:23.545808 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-36a48a73-52cd-41c5-b987-bfd7fc74224b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:23.613944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:23.623107 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 549/1079] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:14:23.903729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460bc51f-8cf2-4bf6-a3e2-d93e2a2a4c88 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:24.010046 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460bc51f-8cf2-4bf6-a3e2-d93e2a2a4c88 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:24.042867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5c0b98-4d17-4305-ac97-5bebeecd48a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:24.043495 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5c0b98-4d17-4305-ac97-5bebeecd48a2 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:24.043771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5c0b98-4d17-4305-ac97-5bebeecd48a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:24.044042 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5c0b98-4d17-4305-ac97-5bebeecd48a2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:24.054477 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5c0b98-4d17-4305-ac97-5bebeecd48a2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:14:24.060172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-3f5c0b98-4d17-4305-ac97-5bebeecd48a2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:14:24.060172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:24.060172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:24.060172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:24.060172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:24.060172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:24.060172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:24.060172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:24.060172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:24.060172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:24.060172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:24.060172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:24.060172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:14:24.060172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:14:24.060172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:14:24.060172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:14:24.061138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:14:24.061138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:14:24.061138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:14:24.061138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:14:24.061138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:24.061138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:24.061138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:24.061138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:24.061138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:24.061138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:24.061138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:24.061138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:24.061138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:24.061138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:24.061138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:24.061138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:24.061992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:24.061992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:24.061992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:24.061992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:24.061992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:14:24.061992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:14:24.061992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:14:24.061992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:14:24.061992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:14:24.061992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:14:24.061992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:14:24.061992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:24.065189 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 531/1080] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:14:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:24.067860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a48a73-52cd-41c5-b987-bfd7fc74224b None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:24.171525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a48a73-52cd-41c5-b987-bfd7fc74224b None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:24.205053 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17202f97-9c90-4953-86a8-5247b4e78952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:24.205541 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17202f97-9c90-4953-86a8-5247b4e78952 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:24.205820 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17202f97-9c90-4953-86a8-5247b4e78952 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:24.206094 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17202f97-9c90-4953-86a8-5247b4e78952 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:24.207301 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17202f97-9c90-4953-86a8-5247b4e78952 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:24.207597 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17202f97-9c90-4953-86a8-5247b4e78952 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:24.208960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17202f97-9c90-4953-86a8-5247b4e78952 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:24.217711 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 550/1081] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:14:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:24.224870 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5c0b98-4d17-4305-ac97-5bebeecd48a2 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:24.254351 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:24.327203 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5c0b98-4d17-4305-ac97-5bebeecd48a2 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:24.363517 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7562d5a2-cba3-46a5-a544-419780822388 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:24.364148 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7562d5a2-cba3-46a5-a544-419780822388 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:24.364428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7562d5a2-cba3-46a5-a544-419780822388 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:24.364702 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7562d5a2-cba3-46a5-a544-419780822388 None admin] PATH_INFO: `/v3/users/8e5773379fb44ac98c91dd141dcdbeac` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:24.365865 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7562d5a2-cba3-46a5-a544-419780822388 None admin] RBAC: Authorizing `identity:get_user(user_id=8e5773379fb44ac98c91dd141dcdbeac)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:24.367315 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7562d5a2-cba3-46a5-a544-419780822388 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:24.368922 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 532/1082] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:14:24 2022] GET /identity/v3/users/8e5773379fb44ac98c91dd141dcdbeac => generated 285 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:24.374870 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17202f97-9c90-4953-86a8-5247b4e78952 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:24.432645 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:24.480044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17202f97-9c90-4953-86a8-5247b4e78952 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:24.514607 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e415c07-a057-4943-bf02-f854dc551c11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:24.516964 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e415c07-a057-4943-bf02-f854dc551c11 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:24.516964 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e415c07-a057-4943-bf02-f854dc551c11 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:24.516964 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e415c07-a057-4943-bf02-f854dc551c11 None admin] PATH_INFO: `/v3/projects/f9ffd2c237014adcabcbab50d24d6134` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:24.517472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e415c07-a057-4943-bf02-f854dc551c11 None admin] RBAC: Authorizing `identity:get_project(project_id=f9ffd2c237014adcabcbab50d24d6134)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:24.518895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e415c07-a057-4943-bf02-f854dc551c11 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:24.520548 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 551/1083] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:14:24 2022] GET /identity/v3/projects/f9ffd2c237014adcabcbab50d24d6134 => generated 311 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:24.526715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7562d5a2-cba3-46a5-a544-419780822388 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:24.629667 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:24.636580 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7562d5a2-cba3-46a5-a544-419780822388 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:24.670899 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e7ca5e-9c4c-493b-b2d5-e2f17ac6d1bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:24.671645 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e7ca5e-9c4c-493b-b2d5-e2f17ac6d1bd None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:24.671917 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e7ca5e-9c4c-493b-b2d5-e2f17ac6d1bd None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:24.672188 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e7ca5e-9c4c-493b-b2d5-e2f17ac6d1bd None admin] PATH_INFO: `/v3/projects/f9ffd2c237014adcabcbab50d24d6134/users/8e5773379fb44ac98c91dd141dcdbeac/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:24.674104 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e7ca5e-9c4c-493b-b2d5-e2f17ac6d1bd None admin] RBAC: Authorizing `identity:create_grant(project_id=f9ffd2c237014adcabcbab50d24d6134, user_id=8e5773379fb44ac98c91dd141dcdbeac, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:24.675550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e7ca5e-9c4c-493b-b2d5-e2f17ac6d1bd None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:24.688421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 533/1084] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:14:24 2022] PUT /identity/v3/projects/f9ffd2c237014adcabcbab50d24d6134/users/8e5773379fb44ac98c91dd141dcdbeac/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:14:25.455671 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653b52be-9266-41c5-bbf3-bd8c29973b9d None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:25.456186 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653b52be-9266-41c5-bbf3-bd8c29973b9d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:25.456295 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653b52be-9266-41c5-bbf3-bd8c29973b9d None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:25.458102 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 552/1085] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:25.463370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2058e2f5-d9c0-4e7b-bb3d-8ba03cb64629 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:25.463808 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2058e2f5-d9c0-4e7b-bb3d-8ba03cb64629 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:25.463899 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2058e2f5-d9c0-4e7b-bb3d-8ba03cb64629 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:25.482343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-2058e2f5-d9c0-4e7b-bb3d-8ba03cb64629 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:25.483524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2058e2f5-d9c0-4e7b-bb3d-8ba03cb64629 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:25.551439 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 534/1086] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:25 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:25.817830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653b52be-9266-41c5-bbf3-bd8c29973b9d None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:25.918644 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653b52be-9266-41c5-bbf3-bd8c29973b9d None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:25.942038 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:25.953704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26eb12ad-e5cb-4810-b76f-dbbe384cafe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:25.954320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26eb12ad-e5cb-4810-b76f-dbbe384cafe3 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:25.954594 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26eb12ad-e5cb-4810-b76f-dbbe384cafe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:25.954859 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26eb12ad-e5cb-4810-b76f-dbbe384cafe3 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:25.965506 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26eb12ad-e5cb-4810-b76f-dbbe384cafe3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:14:25.970397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-26eb12ad-e5cb-4810-b76f-dbbe384cafe3 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 04:14:25.970397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:25.970397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:25.970397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:25.970397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:25.970397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:25.970397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:25.970397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:25.970397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:25.970397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:25.970397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:25.970397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:25.970397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:14:25.970397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:14:25.970397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:14:25.970397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:14:25.971413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:14:25.971413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:14:25.971413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:14:25.971413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:14:25.971413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:25.971413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:25.971413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:25.971413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:25.971413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:25.971413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:25.971413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:25.971413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:25.971413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:25.971413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:25.971413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:25.971413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:25.972807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:25.972807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:25.972807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:25.972807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:25.972807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:14:25.972807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:14:25.972807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:14:25.972807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:14:25.972807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:14:25.972807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:14:25.972807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 04:14:25.972807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:25.972807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 553/1087] 10.222.0.85 () {56 vars in 1120 bytes} [Tue Aug 30 04:14:25 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:25.978308 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2058e2f5-d9c0-4e7b-bb3d-8ba03cb64629 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:26.061656 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:26.079244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2058e2f5-d9c0-4e7b-bb3d-8ba03cb64629 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:26.114175 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66ba5e4-1035-4689-b084-14c3f8a0ebe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:26.114793 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66ba5e4-1035-4689-b084-14c3f8a0ebe0 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:26.115072 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66ba5e4-1035-4689-b084-14c3f8a0ebe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:26.115345 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66ba5e4-1035-4689-b084-14c3f8a0ebe0 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:26.118583 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66ba5e4-1035-4689-b084-14c3f8a0ebe0 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:26.118583 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66ba5e4-1035-4689-b084-14c3f8a0ebe0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:26.118583 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66ba5e4-1035-4689-b084-14c3f8a0ebe0 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:26.126639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 535/1088] 10.222.0.85 () {56 vars in 1115 bytes} [Tue Aug 30 04:14:25 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:26.132568 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26eb12ad-e5cb-4810-b76f-dbbe384cafe3 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:26.234846 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26eb12ad-e5cb-4810-b76f-dbbe384cafe3 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:26.267393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfad6af3-3930-4579-bcc3-95e81db41e8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:26.268024 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfad6af3-3930-4579-bcc3-95e81db41e8f None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:26.268301 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfad6af3-3930-4579-bcc3-95e81db41e8f None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:26.268571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfad6af3-3930-4579-bcc3-95e81db41e8f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:26.279494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfad6af3-3930-4579-bcc3-95e81db41e8f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:26.280927 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfad6af3-3930-4579-bcc3-95e81db41e8f None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:26.295325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-cfad6af3-3930-4579-bcc3-95e81db41e8f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:26.295325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:26.295325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:26.295325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:26.295325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:26.295325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:26.295325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:26.295325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:26.295325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:26.295325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:26.295325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:26.295325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:26.295325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:14:26.295325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:14:26.295325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:14:26.295325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:14:26.296269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:26.296269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:26.296269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:26.296269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:26.296269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:26.296269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:26.296269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:26.296269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:26.296269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:26.296269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:26.296269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:26.296269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:26.296269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:26.296269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:26.296269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:26.296269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:26.296269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:14:26.297205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:14:26.297205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:26.297205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:26.297205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 554/1089] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:14:26.302528 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66ba5e4-1035-4689-b084-14c3f8a0ebe0 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:26.400950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66ba5e4-1035-4689-b084-14c3f8a0ebe0 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:26.434086 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cebe71-7b15-43d5-baaf-53375c6938f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:26.434697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cebe71-7b15-43d5-baaf-53375c6938f0 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:26.434978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cebe71-7b15-43d5-baaf-53375c6938f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:26.435258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cebe71-7b15-43d5-baaf-53375c6938f0 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:26.435903 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cebe71-7b15-43d5-baaf-53375c6938f0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:26.436209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cebe71-7b15-43d5-baaf-53375c6938f0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:26.437638 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cebe71-7b15-43d5-baaf-53375c6938f0 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:26.449691 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 536/1090] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:14:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:26.455556 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfad6af3-3930-4579-bcc3-95e81db41e8f None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:26.491244 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:26.556127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfad6af3-3930-4579-bcc3-95e81db41e8f None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:26.574341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed165d4-37c3-49b2-9884-15a334ccf570 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:26.575027 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed165d4-37c3-49b2-9884-15a334ccf570 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:26.575027 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed165d4-37c3-49b2-9884-15a334ccf570 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:26.576220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 537/1091] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:26.581137 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4734f91-39d0-413b-8ca5-58a28d85402a None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:26.581447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4734f91-39d0-413b-8ca5-58a28d85402a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:26.582249 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4734f91-39d0-413b-8ca5-58a28d85402a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:26.594741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51aba93e-e576-4f74-8db4-8e78c33b03f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:26.595538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51aba93e-e576-4f74-8db4-8e78c33b03f4 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:26.595748 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51aba93e-e576-4f74-8db4-8e78c33b03f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:26.596017 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51aba93e-e576-4f74-8db4-8e78c33b03f4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:26.600335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-e4734f91-39d0-413b-8ca5-58a28d85402a None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:26.601576 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e4734f91-39d0-413b-8ca5-58a28d85402a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:26.603656 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:26.603859 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51aba93e-e576-4f74-8db4-8e78c33b03f4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:26.605320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51aba93e-e576-4f74-8db4-8e78c33b03f4 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:26.616971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-51aba93e-e576-4f74-8db4-8e78c33b03f4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:14:26.616971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:26.616971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:26.616971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:26.616971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:26.616971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:26.616971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:26.616971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:26.616971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:26.616971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:26.616971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:26.616971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:26.616971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:14:26.616971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:14:26.616971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:14:26.616971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:14:26.618058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:26.618058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:26.618058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:14:26.618058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:14:26.618058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:14:26.618058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:14:26.618058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:26.618058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:26.618058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:26.618058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:26.618058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:26.618058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:26.618058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:26.618058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:26.618058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:26.618058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:26.618058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:26.618058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:26.619018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:26.619018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:26.619018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:14:26.619018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:14:26.619018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:14:26.619018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:14:26.619018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:14:26.619018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:14:26.619018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 04:14:26.619018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:26.619018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 555/1092] 10.222.0.85 () {56 vars in 1093 bytes} [Tue Aug 30 04:14:26 2022] GET /identity/v3/users/nova => generated 82 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:26.623809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51aba93e-e576-4f74-8db4-8e78c33b03f4 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:26.671060 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 538/1093] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:14:26.717412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:26.723603 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51aba93e-e576-4f74-8db4-8e78c33b03f4 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:26.757179 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5519df-2972-4d79-9925-864a9df76688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:26.757786 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5519df-2972-4d79-9925-864a9df76688 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:26.758068 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5519df-2972-4d79-9925-864a9df76688 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:26.758342 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5519df-2972-4d79-9925-864a9df76688 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:26.759065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5519df-2972-4d79-9925-864a9df76688 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:26.759371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5519df-2972-4d79-9925-864a9df76688 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:26.760801 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5519df-2972-4d79-9925-864a9df76688 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:26.778745 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 556/1094] 10.222.0.85 () {56 vars in 1133 bytes} [Tue Aug 30 04:14:26 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:26.784460 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4734f91-39d0-413b-8ca5-58a28d85402a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:26.882225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4734f91-39d0-413b-8ca5-58a28d85402a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:26.918701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d452c1a-fd94-46d1-9fa6-3a9261d5030e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:26.919310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d452c1a-fd94-46d1-9fa6-3a9261d5030e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:26.919589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d452c1a-fd94-46d1-9fa6-3a9261d5030e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:26.919861 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d452c1a-fd94-46d1-9fa6-3a9261d5030e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:26.928441 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:26.945577 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5519df-2972-4d79-9925-864a9df76688 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:26.989354 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d452c1a-fd94-46d1-9fa6-3a9261d5030e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:26.990839 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d452c1a-fd94-46d1-9fa6-3a9261d5030e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:27.045632 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5519df-2972-4d79-9925-864a9df76688 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:27.078887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618fe99b-7c9d-45db-91e6-4f8197386dc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:27.079642 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618fe99b-7c9d-45db-91e6-4f8197386dc4 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:27.079918 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618fe99b-7c9d-45db-91e6-4f8197386dc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:27.080181 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618fe99b-7c9d-45db-91e6-4f8197386dc4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:27.090402 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618fe99b-7c9d-45db-91e6-4f8197386dc4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:14:27.095159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-618fe99b-7c9d-45db-91e6-4f8197386dc4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:14:27.095159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:27.095159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:27.095159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:27.095159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:27.095159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:27.095159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:27.095159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:27.095159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:27.095159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:27.095159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:27.095159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:27.095159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:14:27.095159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:14:27.095159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:14:27.095159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:14:27.096093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:14:27.096093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:14:27.096093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:14:27.096093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:14:27.096093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:27.096093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:27.096093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:27.096093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:27.096093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:27.096093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:27.096093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:27.096093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:27.096093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:27.096093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:27.096093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:27.096093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:27.096945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:27.096945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:27.096945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:27.096945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:27.096945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:14:27.096945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:14:27.096945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:14:27.096945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:14:27.096945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:14:27.096945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:14:27.096945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:14:27.096945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:27.096945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 557/1095] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:14:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:14:27.101320 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 539/1096] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:14:26 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 04:14:27.102152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618fe99b-7c9d-45db-91e6-4f8197386dc4 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:27.104977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d452c1a-fd94-46d1-9fa6-3a9261d5030e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:27.120465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:27.147896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:27.204800 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618fe99b-7c9d-45db-91e6-4f8197386dc4 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:27.208532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d452c1a-fd94-46d1-9fa6-3a9261d5030e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:27.239434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9616d1-954e-47ef-9d7a-728e8f3de573 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:27.240033 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9616d1-954e-47ef-9d7a-728e8f3de573 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:27.240307 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9616d1-954e-47ef-9d7a-728e8f3de573 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:27.240573 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9616d1-954e-47ef-9d7a-728e8f3de573 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:27.241263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9616d1-954e-47ef-9d7a-728e8f3de573 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:27.241563 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9616d1-954e-47ef-9d7a-728e8f3de573 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:27.242908 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9616d1-954e-47ef-9d7a-728e8f3de573 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:27.245014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050f6c9f-7577-4040-a6ce-dd6ff0aa2499 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:27.245668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050f6c9f-7577-4040-a6ce-dd6ff0aa2499 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:27.245948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050f6c9f-7577-4040-a6ce-dd6ff0aa2499 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:27.246222 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050f6c9f-7577-4040-a6ce-dd6ff0aa2499 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:27.250853 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 558/1097] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:14:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:27.257001 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050f6c9f-7577-4040-a6ce-dd6ff0aa2499 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:27.257358 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9616d1-954e-47ef-9d7a-728e8f3de573 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:27.258462 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050f6c9f-7577-4040-a6ce-dd6ff0aa2499 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:27.274022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-050f6c9f-7577-4040-a6ce-dd6ff0aa2499 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:27.274022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:27.274022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:27.274022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:27.274022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:27.274022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:27.274022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:27.274022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:27.274022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:27.274022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:27.274022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:27.274022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:27.274022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:14:27.274022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:14:27.274022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:14:27.274022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:14:27.274990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:27.274990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:27.274990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:27.274990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:27.274990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:27.274990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:27.274990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:27.274990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:27.274990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:27.274990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:27.274990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:27.274990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:27.274990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:27.274990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:27.274990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:27.274990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:27.274990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:14:27.280964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:14:27.280964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:14:27.280964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:27.280964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 540/1098] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:14:27.280964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050f6c9f-7577-4040-a6ce-dd6ff0aa2499 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:27.334048 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:27.356404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9616d1-954e-47ef-9d7a-728e8f3de573 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:27.371686 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:27.383001 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050f6c9f-7577-4040-a6ce-dd6ff0aa2499 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:27.390488 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e55978-d959-42e3-834d-3c60ad55c661 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:27.391111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e55978-d959-42e3-834d-3c60ad55c661 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:27.391389 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e55978-d959-42e3-834d-3c60ad55c661 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:27.391662 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e55978-d959-42e3-834d-3c60ad55c661 None admin] PATH_INFO: `/v3/users/8e5773379fb44ac98c91dd141dcdbeac` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:27.393109 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e55978-d959-42e3-834d-3c60ad55c661 None admin] RBAC: Authorizing `identity:get_user(user_id=8e5773379fb44ac98c91dd141dcdbeac)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:27.395101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e55978-d959-42e3-834d-3c60ad55c661 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:27.396675 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 559/1099] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:14:27 2022] GET /identity/v3/users/8e5773379fb44ac98c91dd141dcdbeac => generated 285 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:27.402453 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e55978-d959-42e3-834d-3c60ad55c661 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:27.420105 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e529ff4-70f8-4bac-82ae-517279e5a5ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:27.420715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e529ff4-70f8-4bac-82ae-517279e5a5ab None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:27.420992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e529ff4-70f8-4bac-82ae-517279e5a5ab None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:27.421296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e529ff4-70f8-4bac-82ae-517279e5a5ab None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:27.421938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e529ff4-70f8-4bac-82ae-517279e5a5ab None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:27.422262 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e529ff4-70f8-4bac-82ae-517279e5a5ab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:27.423630 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e529ff4-70f8-4bac-82ae-517279e5a5ab None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:27.435182 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 541/1100] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:14:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:27.441416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e529ff4-70f8-4bac-82ae-517279e5a5ab None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:27.502667 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e55978-d959-42e3-834d-3c60ad55c661 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:27.535831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbad092e-65cd-4934-b509-220676625d09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:27.536426 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbad092e-65cd-4934-b509-220676625d09 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:27.536705 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbad092e-65cd-4934-b509-220676625d09 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:27.536976 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbad092e-65cd-4934-b509-220676625d09 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:27.537112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e529ff4-70f8-4bac-82ae-517279e5a5ab None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:27.551715 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:27.571364 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bbea29-a3d2-48b2-8857-8bddb9161e08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:27.572112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bbea29-a3d2-48b2-8857-8bddb9161e08 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:27.572395 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bbea29-a3d2-48b2-8857-8bddb9161e08 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:27.572672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bbea29-a3d2-48b2-8857-8bddb9161e08 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:27.579251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bbea29-a3d2-48b2-8857-8bddb9161e08 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:27.580699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bbea29-a3d2-48b2-8857-8bddb9161e08 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:27.593075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-d2bbea29-a3d2-48b2-8857-8bddb9161e08 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:14:27.593075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:27.593075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:27.593075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:27.593075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:27.593075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:27.593075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:27.593075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:27.593075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:27.593075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:27.593075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:27.593075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:27.593075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:14:27.593075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:14:27.593075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:14:27.594006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:14:27.594006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:27.594006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:27.594006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:27.594006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:27.594006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:27.594006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:27.594006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:27.594006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:27.594006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:27.594006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:27.594006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:27.594006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:27.594006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:27.594006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:27.594006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:27.594006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:27.595227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:14:27.595227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:14:27.595227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:14:27.595227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:14:27.595227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:14:27.595227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:14:27.595227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:14:27.595227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:27.595227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 542/1101] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:14:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:14:27.600375 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bbea29-a3d2-48b2-8857-8bddb9161e08 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:27.606375 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbad092e-65cd-4934-b509-220676625d09 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:27.607807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbad092e-65cd-4934-b509-220676625d09 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:27.611864 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:27.701168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bbea29-a3d2-48b2-8857-8bddb9161e08 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:27.710781 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 560/1102] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:14:27 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 04:14:27.716619 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbad092e-65cd-4934-b509-220676625d09 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:27.736752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d6b5a4-f6c5-43ee-b657-7c11f13b88cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:27.737377 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d6b5a4-f6c5-43ee-b657-7c11f13b88cd None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:27.737662 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d6b5a4-f6c5-43ee-b657-7c11f13b88cd None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:27.737918 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d6b5a4-f6c5-43ee-b657-7c11f13b88cd None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:27.738605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d6b5a4-f6c5-43ee-b657-7c11f13b88cd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:27.738904 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d6b5a4-f6c5-43ee-b657-7c11f13b88cd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:27.740321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d6b5a4-f6c5-43ee-b657-7c11f13b88cd None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:27.752129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 543/1103] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:14:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:27.758390 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d6b5a4-f6c5-43ee-b657-7c11f13b88cd None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:27.761892 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:27.814845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbad092e-65cd-4934-b509-220676625d09 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:27.830947 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:27.848259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2f53e9-96ac-425a-80d1-b7f41263f47b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:27.848891 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2f53e9-96ac-425a-80d1-b7f41263f47b None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:27.849202 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2f53e9-96ac-425a-80d1-b7f41263f47b None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:27.849474 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2f53e9-96ac-425a-80d1-b7f41263f47b None admin] PATH_INFO: `/v3/projects/f9ffd2c237014adcabcbab50d24d6134` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:27.850584 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2f53e9-96ac-425a-80d1-b7f41263f47b None admin] RBAC: Authorizing `identity:get_project(project_id=f9ffd2c237014adcabcbab50d24d6134)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:27.852009 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2f53e9-96ac-425a-80d1-b7f41263f47b None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:27.853719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 561/1104] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:14:27 2022] GET /identity/v3/projects/f9ffd2c237014adcabcbab50d24d6134 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:27.859805 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2f53e9-96ac-425a-80d1-b7f41263f47b None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:27.865120 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d6b5a4-f6c5-43ee-b657-7c11f13b88cd None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:27.899297 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc64675-aeba-4f83-b448-d3f478badeae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:27.899891 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc64675-aeba-4f83-b448-d3f478badeae None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:27.900237 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc64675-aeba-4f83-b448-d3f478badeae None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:27.900514 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc64675-aeba-4f83-b448-d3f478badeae None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:27.901257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc64675-aeba-4f83-b448-d3f478badeae None admin] RBAC: Adding query filter params (role.id=6b76b888c73142da957a10d32ce5e995, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=4bc4d455ebf041d488b1199ca58358a4) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:27.901565 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc64675-aeba-4f83-b448-d3f478badeae None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:27.902981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc64675-aeba-4f83-b448-d3f478badeae None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:27.908701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 544/1105] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:14:27 2022] GET /identity/v3/role_assignments?user.id=4bc4d455ebf041d488b1199ca58358a4&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=6b76b888c73142da957a10d32ce5e995 => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:27.960621 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2f53e9-96ac-425a-80d1-b7f41263f47b None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:27.975152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:27.998743 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b4ec81-e80e-404e-9a6d-765d3f8f4933 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:27.999359 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b4ec81-e80e-404e-9a6d-765d3f8f4933 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:27.999640 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b4ec81-e80e-404e-9a6d-765d3f8f4933 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:27.999912 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b4ec81-e80e-404e-9a6d-765d3f8f4933 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:28.000742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b4ec81-e80e-404e-9a6d-765d3f8f4933 None admin] RBAC: Adding query filter params (role.id=d17954e7ce544f0d90a512340232c60e, scope.project.id=f9ffd2c237014adcabcbab50d24d6134, user.id=8e5773379fb44ac98c91dd141dcdbeac) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:28.001098 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b4ec81-e80e-404e-9a6d-765d3f8f4933 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:28.002576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b4ec81-e80e-404e-9a6d-765d3f8f4933 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:28.010052 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 562/1106] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:14:27 2022] GET /identity/v3/role_assignments?user.id=8e5773379fb44ac98c91dd141dcdbeac&scope.project.id=f9ffd2c237014adcabcbab50d24d6134&role.id=d17954e7ce544f0d90a512340232c60e => generated 608 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:29.943023 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40f1dda-e7e7-40ef-817e-1e560745baaa None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:29.943562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40f1dda-e7e7-40ef-817e-1e560745baaa None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:29.943562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40f1dda-e7e7-40ef-817e-1e560745baaa None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:29.944891 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 545/1107] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:29.950193 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c51ee5-30b9-444f-bca7-5edfef7e3b9c None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:29.950623 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c51ee5-30b9-444f-bca7-5edfef7e3b9c None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:29.950716 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c51ee5-30b9-444f-bca7-5edfef7e3b9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:29.968278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-47c51ee5-30b9-444f-bca7-5edfef7e3b9c None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:29.969478 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-47c51ee5-30b9-444f-bca7-5edfef7e3b9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:30.033062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 563/1108] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:14:30.312659 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40f1dda-e7e7-40ef-817e-1e560745baaa None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:30.413512 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40f1dda-e7e7-40ef-817e-1e560745baaa None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:30.445840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90e8c7f-fedb-4c4a-948c-3a0077e0490c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:30.446458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90e8c7f-fedb-4c4a-948c-3a0077e0490c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:30.446736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90e8c7f-fedb-4c4a-948c-3a0077e0490c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:30.447008 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90e8c7f-fedb-4c4a-948c-3a0077e0490c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:30.448102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90e8c7f-fedb-4c4a-948c-3a0077e0490c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:30.449632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90e8c7f-fedb-4c4a-948c-3a0077e0490c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:30.451177 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 546/1109] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:30.456936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c51ee5-30b9-444f-bca7-5edfef7e3b9c None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:30.554228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c51ee5-30b9-444f-bca7-5edfef7e3b9c None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:30.561296 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:30.587307 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3a7a15-e540-4cfb-a016-49a010ee74ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:30.587842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3a7a15-e540-4cfb-a016-49a010ee74ec None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:30.588124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3a7a15-e540-4cfb-a016-49a010ee74ec None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:30.588396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3a7a15-e540-4cfb-a016-49a010ee74ec None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:30.589072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3a7a15-e540-4cfb-a016-49a010ee74ec None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:30.590500 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3a7a15-e540-4cfb-a016-49a010ee74ec None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:30.611694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 564/1110] 10.222.0.85 () {60 vars in 1132 bytes} [Tue Aug 30 04:14:30 2022] POST /identity/v3/users => generated 286 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:14:32.559445 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f000877-27f7-4064-b15b-cfbd69ec5624 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:32.559973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f000877-27f7-4064-b15b-cfbd69ec5624 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:32.559973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f000877-27f7-4064-b15b-cfbd69ec5624 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:32.561376 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 547/1111] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:32.566888 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a4b924-a4e7-47a1-a87d-ca673e214b20 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:32.567325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a4b924-a4e7-47a1-a87d-ca673e214b20 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:32.567420 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a4b924-a4e7-47a1-a87d-ca673e214b20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:32.585780 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-31a4b924-a4e7-47a1-a87d-ca673e214b20 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:32.586949 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-31a4b924-a4e7-47a1-a87d-ca673e214b20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:32.651831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 565/1112] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:32.919714 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f000877-27f7-4064-b15b-cfbd69ec5624 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:32.922547 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:33.022216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f000877-27f7-4064-b15b-cfbd69ec5624 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:33.054519 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0669d7eb-8230-40ca-9501-8a7a481d94c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:33.055130 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0669d7eb-8230-40ca-9501-8a7a481d94c7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:33.055404 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0669d7eb-8230-40ca-9501-8a7a481d94c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:33.055671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0669d7eb-8230-40ca-9501-8a7a481d94c7 None admin] PATH_INFO: `/v3/roles/a9e862ac528245ca8363bccc5bf35287` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:33.057032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0669d7eb-8230-40ca-9501-8a7a481d94c7 None admin] RBAC: Authorizing `identity:get_role(role_id=a9e862ac528245ca8363bccc5bf35287)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:33.058422 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0669d7eb-8230-40ca-9501-8a7a481d94c7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:33.059733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 548/1113] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:14:32 2022] GET /identity/v3/roles/a9e862ac528245ca8363bccc5bf35287 => generated 223 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:33.065625 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a4b924-a4e7-47a1-a87d-ca673e214b20 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:33.154848 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:33.165304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a4b924-a4e7-47a1-a87d-ca673e214b20 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:33.198316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875b7f61-eddd-470f-a97a-7b7e6aab9e76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:33.198932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875b7f61-eddd-470f-a97a-7b7e6aab9e76 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:33.199209 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875b7f61-eddd-470f-a97a-7b7e6aab9e76 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:33.199479 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875b7f61-eddd-470f-a97a-7b7e6aab9e76 None admin] PATH_INFO: `/v3/users/5a4edb84e52a430c92c6ee1bb9b6142a` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:33.212774 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875b7f61-eddd-470f-a97a-7b7e6aab9e76 None admin] RBAC: Authorizing `identity:get_user(user_id=5a4edb84e52a430c92c6ee1bb9b6142a)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:33.214246 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875b7f61-eddd-470f-a97a-7b7e6aab9e76 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:33.215871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 566/1114] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:14:33 2022] GET /identity/v3/users/5a4edb84e52a430c92c6ee1bb9b6142a => generated 286 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:33.221613 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0669d7eb-8230-40ca-9501-8a7a481d94c7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:33.290941 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:33.322117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0669d7eb-8230-40ca-9501-8a7a481d94c7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:33.354770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d67230a-ea79-43af-be99-ac974a46a79e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:33.355377 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d67230a-ea79-43af-be99-ac974a46a79e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:33.355653 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d67230a-ea79-43af-be99-ac974a46a79e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:33.355922 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d67230a-ea79-43af-be99-ac974a46a79e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:33.421778 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d67230a-ea79-43af-be99-ac974a46a79e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:33.423221 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d67230a-ea79-43af-be99-ac974a46a79e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:33.489544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:33.520271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 549/1115] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:14:33 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 04:14:33.526193 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875b7f61-eddd-470f-a97a-7b7e6aab9e76 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:33.625162 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875b7f61-eddd-470f-a97a-7b7e6aab9e76 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:33.659036 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f163f638-0c69-4e47-af28-ad640783257a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:33.659746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f163f638-0c69-4e47-af28-ad640783257a None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:33.660040 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f163f638-0c69-4e47-af28-ad640783257a None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:33.660318 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f163f638-0c69-4e47-af28-ad640783257a None admin] PATH_INFO: `/v3/projects/f9ffd2c237014adcabcbab50d24d6134` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:33.661472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f163f638-0c69-4e47-af28-ad640783257a None admin] RBAC: Authorizing `identity:get_project(project_id=f9ffd2c237014adcabcbab50d24d6134)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:33.662904 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f163f638-0c69-4e47-af28-ad640783257a None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:33.664528 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 567/1116] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:14:33 2022] GET /identity/v3/projects/f9ffd2c237014adcabcbab50d24d6134 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:33.670694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d67230a-ea79-43af-be99-ac974a46a79e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:33.768168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d67230a-ea79-43af-be99-ac974a46a79e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:33.802224 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8651e9-7e92-4e21-932d-82e393ed1621 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:33.802821 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8651e9-7e92-4e21-932d-82e393ed1621 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:33.803099 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8651e9-7e92-4e21-932d-82e393ed1621 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:33.803371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8651e9-7e92-4e21-932d-82e393ed1621 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:33.804093 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8651e9-7e92-4e21-932d-82e393ed1621 None admin] RBAC: Adding query filter params (role.id=a9e862ac528245ca8363bccc5bf35287, scope.project.id=f9ffd2c237014adcabcbab50d24d6134, user.id=5a4edb84e52a430c92c6ee1bb9b6142a) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:33.804397 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8651e9-7e92-4e21-932d-82e393ed1621 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:33.805842 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8651e9-7e92-4e21-932d-82e393ed1621 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:33.811133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 550/1117] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:14:33 2022] GET /identity/v3/role_assignments?user.id=5a4edb84e52a430c92c6ee1bb9b6142a&scope.project.id=f9ffd2c237014adcabcbab50d24d6134&role.id=a9e862ac528245ca8363bccc5bf35287 => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:35.712923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433f501f-5ae5-4a56-8d92-5cefae9650f1 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:35.713620 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433f501f-5ae5-4a56-8d92-5cefae9650f1 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:35.713620 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433f501f-5ae5-4a56-8d92-5cefae9650f1 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:35.714926 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 568/1118] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:35.720123 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8792823-ed1f-40a4-ba3c-d9e9bba36bd3 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:35.720676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8792823-ed1f-40a4-ba3c-d9e9bba36bd3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:35.720676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8792823-ed1f-40a4-ba3c-d9e9bba36bd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:35.740226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c8792823-ed1f-40a4-ba3c-d9e9bba36bd3 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:35.741474 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c8792823-ed1f-40a4-ba3c-d9e9bba36bd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:35.768554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:35.810536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 551/1119] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:35 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:36.089479 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433f501f-5ae5-4a56-8d92-5cefae9650f1 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:36.110020 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:36.197205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433f501f-5ae5-4a56-8d92-5cefae9650f1 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:36.233370 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b03ab6-1b72-4602-a6a5-1b40ce65f729 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:36.233993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b03ab6-1b72-4602-a6a5-1b40ce65f729 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:36.234282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b03ab6-1b72-4602-a6a5-1b40ce65f729 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:36.234554 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b03ab6-1b72-4602-a6a5-1b40ce65f729 None admin] PATH_INFO: `/v3/roles/a9e862ac528245ca8363bccc5bf35287` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:36.235952 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b03ab6-1b72-4602-a6a5-1b40ce65f729 None admin] RBAC: Authorizing `identity:get_role(role_id=a9e862ac528245ca8363bccc5bf35287)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:36.237345 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b03ab6-1b72-4602-a6a5-1b40ce65f729 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:36.238698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 569/1120] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:14:36 2022] GET /identity/v3/roles/a9e862ac528245ca8363bccc5bf35287 => generated 223 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:36.245686 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8792823-ed1f-40a4-ba3c-d9e9bba36bd3 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:36.358885 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8792823-ed1f-40a4-ba3c-d9e9bba36bd3 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:36.396361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c05583c-da3b-40a3-bce6-79950f3a64a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:36.397060 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c05583c-da3b-40a3-bce6-79950f3a64a9 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:36.397370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c05583c-da3b-40a3-bce6-79950f3a64a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:36.397709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c05583c-da3b-40a3-bce6-79950f3a64a9 None admin] PATH_INFO: `/v3/users/5a4edb84e52a430c92c6ee1bb9b6142a` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:36.398901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c05583c-da3b-40a3-bce6-79950f3a64a9 None admin] RBAC: Authorizing `identity:get_user(user_id=5a4edb84e52a430c92c6ee1bb9b6142a)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:36.400498 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c05583c-da3b-40a3-bce6-79950f3a64a9 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:36.402256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 552/1121] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:14:36 2022] GET /identity/v3/users/5a4edb84e52a430c92c6ee1bb9b6142a => generated 286 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:36.409768 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b03ab6-1b72-4602-a6a5-1b40ce65f729 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:36.494205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:36.509284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b03ab6-1b72-4602-a6a5-1b40ce65f729 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:36.541651 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8294e8c6-abbd-4eb2-bb0e-dc8108f4651b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:36.542275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8294e8c6-abbd-4eb2-bb0e-dc8108f4651b None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:36.542557 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8294e8c6-abbd-4eb2-bb0e-dc8108f4651b None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:36.542829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8294e8c6-abbd-4eb2-bb0e-dc8108f4651b None admin] PATH_INFO: `/v3/projects/f9ffd2c237014adcabcbab50d24d6134` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:36.543922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8294e8c6-abbd-4eb2-bb0e-dc8108f4651b None admin] RBAC: Authorizing `identity:get_project(project_id=f9ffd2c237014adcabcbab50d24d6134)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:36.545380 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8294e8c6-abbd-4eb2-bb0e-dc8108f4651b None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:36.547028 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 570/1122] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:14:36 2022] GET /identity/v3/projects/f9ffd2c237014adcabcbab50d24d6134 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:36.553095 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c05583c-da3b-40a3-bce6-79950f3a64a9 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:36.584112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:36.650436 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c05583c-da3b-40a3-bce6-79950f3a64a9 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:36.683852 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d174f51-19ee-4ab4-9df0-b5129dee63b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:36.684503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d174f51-19ee-4ab4-9df0-b5129dee63b5 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:36.684781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d174f51-19ee-4ab4-9df0-b5129dee63b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:36.685082 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d174f51-19ee-4ab4-9df0-b5129dee63b5 None admin] PATH_INFO: `/v3/projects/f9ffd2c237014adcabcbab50d24d6134/users/5a4edb84e52a430c92c6ee1bb9b6142a/roles/a9e862ac528245ca8363bccc5bf35287` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:36.687195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d174f51-19ee-4ab4-9df0-b5129dee63b5 None admin] RBAC: Authorizing `identity:create_grant(project_id=f9ffd2c237014adcabcbab50d24d6134, user_id=5a4edb84e52a430c92c6ee1bb9b6142a, role_id=a9e862ac528245ca8363bccc5bf35287)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:36.688627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d174f51-19ee-4ab4-9df0-b5129dee63b5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:36.701998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 553/1123] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:14:36 2022] PUT /identity/v3/projects/f9ffd2c237014adcabcbab50d24d6134/users/5a4edb84e52a430c92c6ee1bb9b6142a/roles/a9e862ac528245ca8363bccc5bf35287 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:14:38.636968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a9096d-0115-4c7e-9ab3-22261bc37ef9 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:38.637575 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a9096d-0115-4c7e-9ab3-22261bc37ef9 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:38.637575 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a9096d-0115-4c7e-9ab3-22261bc37ef9 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:38.638852 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 571/1124] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:38.644278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bc6c0e-8680-4183-a24c-581efe92fbb0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:38.644675 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bc6c0e-8680-4183-a24c-581efe92fbb0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:38.644819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bc6c0e-8680-4183-a24c-581efe92fbb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:38.663877 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b1bc6c0e-8680-4183-a24c-581efe92fbb0 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:38.665126 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b1bc6c0e-8680-4183-a24c-581efe92fbb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:38.730549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 554/1125] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:14:39.002325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a9096d-0115-4c7e-9ab3-22261bc37ef9 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:39.107704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a9096d-0115-4c7e-9ab3-22261bc37ef9 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:39.142813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e2c487-ae97-4a19-b9a6-ac62c23890ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:39.143429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e2c487-ae97-4a19-b9a6-ac62c23890ac None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:39.144233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e2c487-ae97-4a19-b9a6-ac62c23890ac None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:39.144501 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e2c487-ae97-4a19-b9a6-ac62c23890ac None admin] PATH_INFO: `/v3/roles/a9e862ac528245ca8363bccc5bf35287` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:39.145889 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e2c487-ae97-4a19-b9a6-ac62c23890ac None admin] RBAC: Authorizing `identity:get_role(role_id=a9e862ac528245ca8363bccc5bf35287)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:39.147251 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e2c487-ae97-4a19-b9a6-ac62c23890ac None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:39.148553 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 572/1126] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:14:39 2022] GET /identity/v3/roles/a9e862ac528245ca8363bccc5bf35287 => generated 223 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:39.154406 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bc6c0e-8680-4183-a24c-581efe92fbb0 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:39.180098 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:39.257084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bc6c0e-8680-4183-a24c-581efe92fbb0 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:39.290180 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2b70f3-a914-4fae-8115-7151eba4b8d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:39.290808 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2b70f3-a914-4fae-8115-7151eba4b8d7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:39.291083 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2b70f3-a914-4fae-8115-7151eba4b8d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:39.291353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2b70f3-a914-4fae-8115-7151eba4b8d7 None admin] PATH_INFO: `/v3/users/5a4edb84e52a430c92c6ee1bb9b6142a` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:39.292444 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e2b70f3-a914-4fae-8115-7151eba4b8d7 None admin] RBAC: Authorizing `identity:get_user(user_id=5a4edb84e52a430c92c6ee1bb9b6142a)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:39.293920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e2b70f3-a914-4fae-8115-7151eba4b8d7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:39.295488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 555/1127] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:14:39 2022] GET /identity/v3/users/5a4edb84e52a430c92c6ee1bb9b6142a => generated 286 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:39.301329 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e2c487-ae97-4a19-b9a6-ac62c23890ac None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:39.325681 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:39.405219 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e2c487-ae97-4a19-b9a6-ac62c23890ac None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:39.438050 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad16e2fa-7e99-4bf9-b28c-42407705b4b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:39.438650 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad16e2fa-7e99-4bf9-b28c-42407705b4b1 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:39.438926 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad16e2fa-7e99-4bf9-b28c-42407705b4b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:39.439195 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad16e2fa-7e99-4bf9-b28c-42407705b4b1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:39.508786 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad16e2fa-7e99-4bf9-b28c-42407705b4b1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:39.510177 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad16e2fa-7e99-4bf9-b28c-42407705b4b1 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:39.532968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:39.606288 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 573/1128] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:14:39 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 04:14:39.612335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2b70f3-a914-4fae-8115-7151eba4b8d7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:39.701160 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:39.711188 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2b70f3-a914-4fae-8115-7151eba4b8d7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:39.744442 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b519434-21a7-455d-a2dd-d0252b8d1ed0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:39.745083 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b519434-21a7-455d-a2dd-d0252b8d1ed0 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:39.745364 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b519434-21a7-455d-a2dd-d0252b8d1ed0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:39.745633 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b519434-21a7-455d-a2dd-d0252b8d1ed0 None admin] PATH_INFO: `/v3/projects/f9ffd2c237014adcabcbab50d24d6134` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:39.746727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b519434-21a7-455d-a2dd-d0252b8d1ed0 None admin] RBAC: Authorizing `identity:get_project(project_id=f9ffd2c237014adcabcbab50d24d6134)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:39.748153 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b519434-21a7-455d-a2dd-d0252b8d1ed0 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:39.749820 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 556/1129] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:14:39 2022] GET /identity/v3/projects/f9ffd2c237014adcabcbab50d24d6134 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:39.756465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad16e2fa-7e99-4bf9-b28c-42407705b4b1 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:39.855329 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad16e2fa-7e99-4bf9-b28c-42407705b4b1 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:39.877725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:39.890382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e280e8bf-2779-4df9-8956-4122fc3fc4db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:39.890981 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e280e8bf-2779-4df9-8956-4122fc3fc4db None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:39.891257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e280e8bf-2779-4df9-8956-4122fc3fc4db None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:39.891526 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e280e8bf-2779-4df9-8956-4122fc3fc4db None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:39.892247 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e280e8bf-2779-4df9-8956-4122fc3fc4db None admin] RBAC: Adding query filter params (role.id=a9e862ac528245ca8363bccc5bf35287, scope.project.id=f9ffd2c237014adcabcbab50d24d6134, user.id=5a4edb84e52a430c92c6ee1bb9b6142a) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:39.892552 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e280e8bf-2779-4df9-8956-4122fc3fc4db None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:39.894042 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e280e8bf-2779-4df9-8956-4122fc3fc4db None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:39.900043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 574/1130] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:14:39 2022] GET /identity/v3/role_assignments?user.id=5a4edb84e52a430c92c6ee1bb9b6142a&scope.project.id=f9ffd2c237014adcabcbab50d24d6134&role.id=a9e862ac528245ca8363bccc5bf35287 => generated 608 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:14:41.839331 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8610c4df-1c94-48e7-989a-28f40dbea8b2 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:41.839895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8610c4df-1c94-48e7-989a-28f40dbea8b2 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:41.839895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8610c4df-1c94-48e7-989a-28f40dbea8b2 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:41.841253 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 557/1131] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:41.846502 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41db28f3-76c3-4831-a901-f3dda65ece8d None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:41.847505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41db28f3-76c3-4831-a901-f3dda65ece8d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:41.847614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41db28f3-76c3-4831-a901-f3dda65ece8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:41.866203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-41db28f3-76c3-4831-a901-f3dda65ece8d None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:41.867375 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-41db28f3-76c3-4831-a901-f3dda65ece8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:41.942440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 575/1132] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:42.220999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8610c4df-1c94-48e7-989a-28f40dbea8b2 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:42.324989 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8610c4df-1c94-48e7-989a-28f40dbea8b2 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:42.359766 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c2d84a-5bdb-4787-9859-403becd6e01b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:42.360399 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c2d84a-5bdb-4787-9859-403becd6e01b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:42.360730 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c2d84a-5bdb-4787-9859-403becd6e01b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:42.360956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c2d84a-5bdb-4787-9859-403becd6e01b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:42.362102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c2d84a-5bdb-4787-9859-403becd6e01b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:42.363888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c2d84a-5bdb-4787-9859-403becd6e01b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:42.365505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 558/1133] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:42.371373 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41db28f3-76c3-4831-a901-f3dda65ece8d None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:42.469211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41db28f3-76c3-4831-a901-f3dda65ece8d None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:42.485717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:42.504171 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8289d77f-d2a0-49ab-b377-cdb306cdb528 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:42.504776 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8289d77f-d2a0-49ab-b377-cdb306cdb528 None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:42.505192 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8289d77f-d2a0-49ab-b377-cdb306cdb528 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:42.505470 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8289d77f-d2a0-49ab-b377-cdb306cdb528 None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:42.506427 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8289d77f-d2a0-49ab-b377-cdb306cdb528 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:42.507817 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8289d77f-d2a0-49ab-b377-cdb306cdb528 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:42.534766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 576/1134] 10.222.0.85 () {60 vars in 1141 bytes} [Tue Aug 30 04:14:42 2022] POST /identity/v3/projects => generated 311 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:14:44.464857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56413cb3-2a32-4526-b48a-2bac7b543254 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:44.465414 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56413cb3-2a32-4526-b48a-2bac7b543254 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:44.465414 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56413cb3-2a32-4526-b48a-2bac7b543254 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:44.466766 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 559/1135] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:44.472635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa4ed90-5d2b-4d88-91a0-f456b4fbc19d None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:44.473535 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa4ed90-5d2b-4d88-91a0-f456b4fbc19d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:44.473535 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa4ed90-5d2b-4d88-91a0-f456b4fbc19d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:44.491600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-0fa4ed90-5d2b-4d88-91a0-f456b4fbc19d None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:44.492766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-0fa4ed90-5d2b-4d88-91a0-f456b4fbc19d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:44.559976 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 577/1136] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:14:44.830595 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56413cb3-2a32-4526-b48a-2bac7b543254 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:44.859125 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:44.936430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56413cb3-2a32-4526-b48a-2bac7b543254 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:44.971819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5feed046-d1f0-4764-9c70-1ea08ac89123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:44.972508 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5feed046-d1f0-4764-9c70-1ea08ac89123 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:44.972791 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5feed046-d1f0-4764-9c70-1ea08ac89123 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:44.973088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5feed046-d1f0-4764-9c70-1ea08ac89123 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:44.974226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5feed046-d1f0-4764-9c70-1ea08ac89123 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:44.975727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5feed046-d1f0-4764-9c70-1ea08ac89123 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:44.977344 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 560/1137] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:14:44 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:44.983248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa4ed90-5d2b-4d88-91a0-f456b4fbc19d None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:45.089977 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:45.094020 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa4ed90-5d2b-4d88-91a0-f456b4fbc19d None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:45.133902 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbf2821-33d5-4f06-8770-681d9b9f5e67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:45.134510 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbf2821-33d5-4f06-8770-681d9b9f5e67 None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:45.134791 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbf2821-33d5-4f06-8770-681d9b9f5e67 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:45.135063 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbf2821-33d5-4f06-8770-681d9b9f5e67 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:45.135725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbf2821-33d5-4f06-8770-681d9b9f5e67 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:45.137190 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbf2821-33d5-4f06-8770-681d9b9f5e67 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:45.158877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 578/1138] 10.222.0.85 () {60 vars in 1132 bytes} [Tue Aug 30 04:14:44 2022] POST /identity/v3/users => generated 286 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:14:47.087360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978cc5eb-eaf9-451b-9174-78b79a6b8f0e None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:47.087937 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978cc5eb-eaf9-451b-9174-78b79a6b8f0e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:47.087937 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978cc5eb-eaf9-451b-9174-78b79a6b8f0e None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:47.089248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 561/1139] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:47.094275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c282464-74af-46df-babc-b81a3d7069b7 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:47.094722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c282464-74af-46df-babc-b81a3d7069b7 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:47.094826 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c282464-74af-46df-babc-b81a3d7069b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:47.122837 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-7c282464-74af-46df-babc-b81a3d7069b7 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:47.124041 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-7c282464-74af-46df-babc-b81a3d7069b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:47.189717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 579/1140] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:47.459306 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978cc5eb-eaf9-451b-9174-78b79a6b8f0e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:47.552051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:47.559818 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978cc5eb-eaf9-451b-9174-78b79a6b8f0e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:47.593341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da39d0ea-1ab8-4072-9ade-1bdda20b3837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:47.593991 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da39d0ea-1ab8-4072-9ade-1bdda20b3837 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:47.594269 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da39d0ea-1ab8-4072-9ade-1bdda20b3837 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:47.594707 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da39d0ea-1ab8-4072-9ade-1bdda20b3837 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:47.604341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da39d0ea-1ab8-4072-9ade-1bdda20b3837 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:14:47.609194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-da39d0ea-1ab8-4072-9ade-1bdda20b3837 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:14:47.609194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:47.609194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:47.609194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:47.609194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:47.609194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:47.609194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:47.609194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:47.609194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:47.609194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:47.609194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:47.609194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:47.609194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:14:47.609194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:14:47.609194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:14:47.609194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:14:47.610341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:14:47.610341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:14:47.610341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:14:47.610341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:14:47.610341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:47.610341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:47.610341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:47.610341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:47.610341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:47.610341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:47.610341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:47.610341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:47.610341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:47.610341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:47.610341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:47.610341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:47.611438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:47.611438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:47.611438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:47.611438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:47.611438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:14:47.611438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:14:47.611438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:14:47.611438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:14:47.611438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:14:47.611438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:14:47.611438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:14:47.611438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:47.611438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 562/1141] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:14:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:47.616427 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c282464-74af-46df-babc-b81a3d7069b7 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:47.698919 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:47.715899 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c282464-74af-46df-babc-b81a3d7069b7 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:47.749433 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53356389-deb0-4f56-a5af-0694721bc789 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:47.750029 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53356389-deb0-4f56-a5af-0694721bc789 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:47.750310 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53356389-deb0-4f56-a5af-0694721bc789 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:47.750581 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53356389-deb0-4f56-a5af-0694721bc789 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:47.751246 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53356389-deb0-4f56-a5af-0694721bc789 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:47.751543 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53356389-deb0-4f56-a5af-0694721bc789 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:47.752932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53356389-deb0-4f56-a5af-0694721bc789 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:47.761046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 580/1142] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:14:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:47.766994 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da39d0ea-1ab8-4072-9ade-1bdda20b3837 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:47.865026 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da39d0ea-1ab8-4072-9ade-1bdda20b3837 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:47.899488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f49344f-0319-415b-99a0-e629d0590bd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:47.900104 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f49344f-0319-415b-99a0-e629d0590bd8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:47.900384 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f49344f-0319-415b-99a0-e629d0590bd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:47.900659 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f49344f-0319-415b-99a0-e629d0590bd8 None admin] PATH_INFO: `/v3/users/60bf9379711c4c6c947dc15dfc9f5da7` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:47.913647 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f49344f-0319-415b-99a0-e629d0590bd8 None admin] RBAC: Authorizing `identity:get_user(user_id=60bf9379711c4c6c947dc15dfc9f5da7)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:47.915088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f49344f-0319-415b-99a0-e629d0590bd8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:47.916668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 563/1143] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:14:47 2022] GET /identity/v3/users/60bf9379711c4c6c947dc15dfc9f5da7 => generated 286 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:47.922849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53356389-deb0-4f56-a5af-0694721bc789 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:48.021810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53356389-deb0-4f56-a5af-0694721bc789 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:48.054497 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91b047d-3ca4-49bd-89b2-5006e32ec584 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:48.055088 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91b047d-3ca4-49bd-89b2-5006e32ec584 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:48.055366 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91b047d-3ca4-49bd-89b2-5006e32ec584 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:48.055641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91b047d-3ca4-49bd-89b2-5006e32ec584 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:48.070785 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:48.122594 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91b047d-3ca4-49bd-89b2-5006e32ec584 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:48.124060 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91b047d-3ca4-49bd-89b2-5006e32ec584 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:48.220412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 581/1144] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:14:47 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 04:14:48.226911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f49344f-0319-415b-99a0-e629d0590bd8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:48.241428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:48.325401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f49344f-0319-415b-99a0-e629d0590bd8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:48.357872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3427c61c-96f5-4ca1-ab8b-5b32f181054c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:48.358495 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3427c61c-96f5-4ca1-ab8b-5b32f181054c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:48.358774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3427c61c-96f5-4ca1-ab8b-5b32f181054c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:48.359045 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3427c61c-96f5-4ca1-ab8b-5b32f181054c None admin] PATH_INFO: `/v3/projects/6d0643a8c3344978831895a456043fb1` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:48.360148 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3427c61c-96f5-4ca1-ab8b-5b32f181054c None admin] RBAC: Authorizing `identity:get_project(project_id=6d0643a8c3344978831895a456043fb1)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:48.361609 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3427c61c-96f5-4ca1-ab8b-5b32f181054c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:48.363214 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 564/1145] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:14:48 2022] GET /identity/v3/projects/6d0643a8c3344978831895a456043fb1 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:48.370093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91b047d-3ca4-49bd-89b2-5006e32ec584 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:48.422780 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:48.475411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91b047d-3ca4-49bd-89b2-5006e32ec584 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:48.509333 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba75a0ab-095b-4d1a-ba30-8b374753d896 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:48.509922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba75a0ab-095b-4d1a-ba30-8b374753d896 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:48.510198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba75a0ab-095b-4d1a-ba30-8b374753d896 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:48.510467 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba75a0ab-095b-4d1a-ba30-8b374753d896 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:48.511175 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba75a0ab-095b-4d1a-ba30-8b374753d896 None admin] RBAC: Adding query filter params (role.id=d17954e7ce544f0d90a512340232c60e, scope.project.id=6d0643a8c3344978831895a456043fb1, user.id=60bf9379711c4c6c947dc15dfc9f5da7) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:48.511476 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba75a0ab-095b-4d1a-ba30-8b374753d896 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:48.512889 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba75a0ab-095b-4d1a-ba30-8b374753d896 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:48.518062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 582/1146] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:14:48 2022] GET /identity/v3/role_assignments?user.id=60bf9379711c4c6c947dc15dfc9f5da7&scope.project.id=6d0643a8c3344978831895a456043fb1&role.id=d17954e7ce544f0d90a512340232c60e => generated 259 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:50.405839 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a9aa23-d51a-44d0-bc9f-9a77f79844a8 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:50.406360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a9aa23-d51a-44d0-bc9f-9a77f79844a8 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:50.406360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a9aa23-d51a-44d0-bc9f-9a77f79844a8 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:50.407729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 565/1147] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:50.413626 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5b297b-6fc7-4bb2-ad46-f96dd6b2369e None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:50.414034 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5b297b-6fc7-4bb2-ad46-f96dd6b2369e None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:50.414179 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5b297b-6fc7-4bb2-ad46-f96dd6b2369e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:50.432646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-0e5b297b-6fc7-4bb2-ad46-f96dd6b2369e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:50.433861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-0e5b297b-6fc7-4bb2-ad46-f96dd6b2369e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:50.502373 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 583/1148] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:50.772986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a9aa23-d51a-44d0-bc9f-9a77f79844a8 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:50.852222 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:50.872268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a9aa23-d51a-44d0-bc9f-9a77f79844a8 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:50.905393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee7cbe7-58c4-41ab-b09e-d2a6577545ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:50.906008 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee7cbe7-58c4-41ab-b09e-d2a6577545ee None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:50.906285 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee7cbe7-58c4-41ab-b09e-d2a6577545ee None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:50.906554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee7cbe7-58c4-41ab-b09e-d2a6577545ee None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:50.916879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee7cbe7-58c4-41ab-b09e-d2a6577545ee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:14:50.921605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-5ee7cbe7-58c4-41ab-b09e-d2a6577545ee None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:14:50.921605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:50.921605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:50.921605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:50.921605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:50.921605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:50.921605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:50.921605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:50.921605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:50.921605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:50.921605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:50.921605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:50.921605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:14:50.921605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:14:50.921605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:14:50.921605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:14:50.922672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:14:50.922672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:14:50.922672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:14:50.922672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:14:50.922672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:50.922672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:50.922672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:50.922672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:50.922672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:50.922672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:50.922672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:50.922672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:50.922672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:50.922672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:50.922672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:50.922672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:50.923539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:50.923539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:50.923539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:50.923539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:50.923539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:14:50.923539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:14:50.923539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:14:50.923539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:14:50.923539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:14:50.923539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:14:50.923539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:14:50.923539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:50.923539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 566/1149] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:14:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:50.928860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5b297b-6fc7-4bb2-ad46-f96dd6b2369e None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:50.968647 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:51.026015 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5b297b-6fc7-4bb2-ad46-f96dd6b2369e None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:51.058797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405327aa-ccdb-4001-8324-c366502929f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:51.059398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405327aa-ccdb-4001-8324-c366502929f2 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:51.059672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405327aa-ccdb-4001-8324-c366502929f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:51.059936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405327aa-ccdb-4001-8324-c366502929f2 None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:51.060585 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405327aa-ccdb-4001-8324-c366502929f2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:51.060880 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405327aa-ccdb-4001-8324-c366502929f2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:51.062257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405327aa-ccdb-4001-8324-c366502929f2 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:51.069977 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 584/1150] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:14:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:51.075766 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee7cbe7-58c4-41ab-b09e-d2a6577545ee None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:51.175426 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee7cbe7-58c4-41ab-b09e-d2a6577545ee None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:51.208799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10c1054-2cec-4757-9101-eee81f49fd71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:51.210583 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10c1054-2cec-4757-9101-eee81f49fd71 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:51.210583 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10c1054-2cec-4757-9101-eee81f49fd71 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:51.211167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10c1054-2cec-4757-9101-eee81f49fd71 None admin] PATH_INFO: `/v3/users/60bf9379711c4c6c947dc15dfc9f5da7` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:51.212030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10c1054-2cec-4757-9101-eee81f49fd71 None admin] RBAC: Authorizing `identity:get_user(user_id=60bf9379711c4c6c947dc15dfc9f5da7)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:51.213497 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10c1054-2cec-4757-9101-eee81f49fd71 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:51.215069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 567/1151] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:14:51 2022] GET /identity/v3/users/60bf9379711c4c6c947dc15dfc9f5da7 => generated 286 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:51.220840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405327aa-ccdb-4001-8324-c366502929f2 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:51.319787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405327aa-ccdb-4001-8324-c366502929f2 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:51.321254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:51.353161 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f638213-c1cb-4429-a1b0-19f00ebf329b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:51.353783 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f638213-c1cb-4429-a1b0-19f00ebf329b None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:51.354063 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f638213-c1cb-4429-a1b0-19f00ebf329b None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:51.354337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f638213-c1cb-4429-a1b0-19f00ebf329b None admin] PATH_INFO: `/v3/projects/6d0643a8c3344978831895a456043fb1` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:51.355442 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f638213-c1cb-4429-a1b0-19f00ebf329b None admin] RBAC: Authorizing `identity:get_project(project_id=6d0643a8c3344978831895a456043fb1)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:51.356877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f638213-c1cb-4429-a1b0-19f00ebf329b None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:51.358529 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 585/1152] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:14:51 2022] GET /identity/v3/projects/6d0643a8c3344978831895a456043fb1 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:51.365122 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10c1054-2cec-4757-9101-eee81f49fd71 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:51.375819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:51.461310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10c1054-2cec-4757-9101-eee81f49fd71 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:51.494436 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b91f13-1263-4d11-b4dc-2a89ef1e4c17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:51.495088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b91f13-1263-4d11-b4dc-2a89ef1e4c17 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:51.495367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b91f13-1263-4d11-b4dc-2a89ef1e4c17 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:51.495640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b91f13-1263-4d11-b4dc-2a89ef1e4c17 None admin] PATH_INFO: `/v3/projects/6d0643a8c3344978831895a456043fb1/users/60bf9379711c4c6c947dc15dfc9f5da7/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:51.497524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b91f13-1263-4d11-b4dc-2a89ef1e4c17 None admin] RBAC: Authorizing `identity:create_grant(project_id=6d0643a8c3344978831895a456043fb1, user_id=60bf9379711c4c6c947dc15dfc9f5da7, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:51.498938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b91f13-1263-4d11-b4dc-2a89ef1e4c17 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:51.512090 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 568/1153] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:14:51 2022] PUT /identity/v3/projects/6d0643a8c3344978831895a456043fb1/users/60bf9379711c4c6c947dc15dfc9f5da7/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:14:53.401397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b890e79-3903-452f-8a7d-fccac51e9b9c None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:53.401913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b890e79-3903-452f-8a7d-fccac51e9b9c None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:53.401913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b890e79-3903-452f-8a7d-fccac51e9b9c None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:53.403203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 586/1154] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:53.408406 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a710993-d8be-439c-a278-4b6100ca7c4e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:53.408802 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a710993-d8be-439c-a278-4b6100ca7c4e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:53.408951 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a710993-d8be-439c-a278-4b6100ca7c4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:53.427352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5a710993-d8be-439c-a278-4b6100ca7c4e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:53.428527 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5a710993-d8be-439c-a278-4b6100ca7c4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:53.488915 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:53.495354 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 569/1155] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:53.763866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b890e79-3903-452f-8a7d-fccac51e9b9c None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:53.874678 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b890e79-3903-452f-8a7d-fccac51e9b9c None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:53.910843 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c321f56-640b-4a9d-8c05-ff4932d33821 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:53.911462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c321f56-640b-4a9d-8c05-ff4932d33821 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:53.911742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c321f56-640b-4a9d-8c05-ff4932d33821 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:53.912012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c321f56-640b-4a9d-8c05-ff4932d33821 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:53.922148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c321f56-640b-4a9d-8c05-ff4932d33821 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:14:53.927447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-1c321f56-640b-4a9d-8c05-ff4932d33821 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:14:53.927447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:53.927447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:53.927447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:53.927447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:53.927447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:53.927447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:53.927447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:53.927447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:53.927447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:53.927447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:53.927447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:53.927447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:14:53.927447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:14:53.927447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:14:53.927447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:14:53.928539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:14:53.928539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:14:53.928539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:14:53.928539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:14:53.928539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:53.928539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:53.928539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:53.928539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:53.928539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:53.928539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:53.928539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:53.928539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:53.928539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:53.928539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:53.928539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:53.928539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:53.929750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:53.929750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:53.929750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:53.929750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:53.929750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:14:53.929750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:14:53.929750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:14:53.929750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:14:53.929750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:14:53.929750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:14:53.929750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:14:53.929750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:14:53.929750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 587/1156] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:14:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:53.934545 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a710993-d8be-439c-a278-4b6100ca7c4e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:54.031038 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a710993-d8be-439c-a278-4b6100ca7c4e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:54.064042 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f61581-1434-4f3e-8c5b-562469955a76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:54.064637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f61581-1434-4f3e-8c5b-562469955a76 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:54.064926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f61581-1434-4f3e-8c5b-562469955a76 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:54.065773 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f61581-1434-4f3e-8c5b-562469955a76 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:54.066442 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f61581-1434-4f3e-8c5b-562469955a76 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:54.066746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f61581-1434-4f3e-8c5b-562469955a76 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:54.068115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f61581-1434-4f3e-8c5b-562469955a76 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:54.076107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 570/1157] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:14:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:54.082038 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c321f56-640b-4a9d-8c05-ff4932d33821 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:54.112604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:54.181888 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c321f56-640b-4a9d-8c05-ff4932d33821 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:54.214430 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8998cbb3-0e23-403c-b269-78cefc86a68b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:54.215041 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8998cbb3-0e23-403c-b269-78cefc86a68b None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:54.215316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8998cbb3-0e23-403c-b269-78cefc86a68b None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:54.215587 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8998cbb3-0e23-403c-b269-78cefc86a68b None admin] PATH_INFO: `/v3/users/60bf9379711c4c6c947dc15dfc9f5da7` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:54.216676 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8998cbb3-0e23-403c-b269-78cefc86a68b None admin] RBAC: Authorizing `identity:get_user(user_id=60bf9379711c4c6c947dc15dfc9f5da7)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:54.218127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8998cbb3-0e23-403c-b269-78cefc86a68b None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:54.219766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 588/1158] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:14:54 2022] GET /identity/v3/users/60bf9379711c4c6c947dc15dfc9f5da7 => generated 286 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:54.225887 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f61581-1434-4f3e-8c5b-562469955a76 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:54.283077 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:54.323970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f61581-1434-4f3e-8c5b-562469955a76 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:54.356290 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3a4ad5-c106-4749-ac3a-76326a3ea22e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:54.356891 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3a4ad5-c106-4749-ac3a-76326a3ea22e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:54.357204 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3a4ad5-c106-4749-ac3a-76326a3ea22e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:54.357480 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3a4ad5-c106-4749-ac3a-76326a3ea22e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:54.422386 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3a4ad5-c106-4749-ac3a-76326a3ea22e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:54.423835 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3a4ad5-c106-4749-ac3a-76326a3ea22e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:54.489433 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:54.529054 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 571/1159] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:14:54 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 04:14:54.535540 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8998cbb3-0e23-403c-b269-78cefc86a68b None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:54.632421 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:54.636460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8998cbb3-0e23-403c-b269-78cefc86a68b None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:54.670873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efaafd49-c320-4c0b-a44d-42d4a6a9695b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:54.671489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efaafd49-c320-4c0b-a44d-42d4a6a9695b None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:54.671764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efaafd49-c320-4c0b-a44d-42d4a6a9695b None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:54.672032 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efaafd49-c320-4c0b-a44d-42d4a6a9695b None admin] PATH_INFO: `/v3/projects/6d0643a8c3344978831895a456043fb1` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:54.673153 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efaafd49-c320-4c0b-a44d-42d4a6a9695b None admin] RBAC: Authorizing `identity:get_project(project_id=6d0643a8c3344978831895a456043fb1)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:54.674570 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efaafd49-c320-4c0b-a44d-42d4a6a9695b None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:54.676172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 589/1160] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:14:54 2022] GET /identity/v3/projects/6d0643a8c3344978831895a456043fb1 => generated 311 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:54.682282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3a4ad5-c106-4749-ac3a-76326a3ea22e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:54.779190 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3a4ad5-c106-4749-ac3a-76326a3ea22e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:54.812599 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58677371-b612-4be9-8acc-58050ba5b8b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:54.813165 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58677371-b612-4be9-8acc-58050ba5b8b4 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:54.813487 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58677371-b612-4be9-8acc-58050ba5b8b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:54.813726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58677371-b612-4be9-8acc-58050ba5b8b4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:54.814447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58677371-b612-4be9-8acc-58050ba5b8b4 None admin] RBAC: Adding query filter params (role.id=d17954e7ce544f0d90a512340232c60e, scope.project.id=6d0643a8c3344978831895a456043fb1, user.id=60bf9379711c4c6c947dc15dfc9f5da7) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:54.814756 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58677371-b612-4be9-8acc-58050ba5b8b4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:54.816748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58677371-b612-4be9-8acc-58050ba5b8b4 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:54.822413 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 572/1161] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:14:54 2022] GET /identity/v3/role_assignments?user.id=60bf9379711c4c6c947dc15dfc9f5da7&scope.project.id=6d0643a8c3344978831895a456043fb1&role.id=d17954e7ce544f0d90a512340232c60e => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:56.724952 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43876115-ce0b-4d91-b9f4-b7a38ac68032 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:56.725539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43876115-ce0b-4d91-b9f4-b7a38ac68032 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:56.725539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43876115-ce0b-4d91-b9f4-b7a38ac68032 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:56.726857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 590/1162] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:56.732093 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fb13b3-008b-435c-8ffa-68e7ede2d81e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:56.732511 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fb13b3-008b-435c-8ffa-68e7ede2d81e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:56.732648 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fb13b3-008b-435c-8ffa-68e7ede2d81e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:56.750318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-43fb13b3-008b-435c-8ffa-68e7ede2d81e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:56.751507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-43fb13b3-008b-435c-8ffa-68e7ede2d81e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:56.769589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:56.817740 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 573/1163] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:56 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:57.090768 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43876115-ce0b-4d91-b9f4-b7a38ac68032 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:57.192996 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43876115-ce0b-4d91-b9f4-b7a38ac68032 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:57.225526 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0fcff8-48da-4d8b-9147-303c0a477c06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:57.226118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0fcff8-48da-4d8b-9147-303c0a477c06 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:57.226394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0fcff8-48da-4d8b-9147-303c0a477c06 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:57.226665 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0fcff8-48da-4d8b-9147-303c0a477c06 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:57.260434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:57.294062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0fcff8-48da-4d8b-9147-303c0a477c06 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:57.295506 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0fcff8-48da-4d8b-9147-303c0a477c06 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:57.391205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 591/1164] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:14:57 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 04:14:57.397214 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43fb13b3-008b-435c-8ffa-68e7ede2d81e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:57.493788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43fb13b3-008b-435c-8ffa-68e7ede2d81e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:57.526275 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a66a32-691b-42ef-b56f-bb17048c875a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:57.526897 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a66a32-691b-42ef-b56f-bb17048c875a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:57.527179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a66a32-691b-42ef-b56f-bb17048c875a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:57.527454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a66a32-691b-42ef-b56f-bb17048c875a None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:57.534418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a66a32-691b-42ef-b56f-bb17048c875a None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:57.535856 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a66a32-691b-42ef-b56f-bb17048c875a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:57.546984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-d5a66a32-691b-42ef-b56f-bb17048c875a None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 30 04:14:57.546984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:57.546984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Aug 30 04:14:57.546984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Aug 30 04:14:57.546984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:14:57.546984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:14:57.546984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:14:57.546984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:14:57.546984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Aug 30 04:14:57.546984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:57.546984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:14:57.546984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:57.546984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:14:57.546984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:14:57.546984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:14:57.546984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:14:57.546984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:14:57.548162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:14:57.548162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:14:57.548162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:14:57.548162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:14:57.548162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:14:57.548162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:14:57.548162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:14:57.548162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:14:57.548162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:14:57.548162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:14:57.548162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:14:57.548162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:14:57.548162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:14:57.548162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:14:57.548162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:14:57.548162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:14:57.549009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:14:57.549009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:14:57.549009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:14:57.549009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:14:57.549009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:14:57.549009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:14:57.549009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:14:57.549009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:14:57.549009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:14:57.549009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:14:57.549009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Aug 30 04:14:57.549009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:14:57.549009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 30 04:14:57.549009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:14:57.549009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 574/1165] 10.222.0.85 () {56 vars in 1117 bytes} [Tue Aug 30 04:14:57 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:14:57.554630 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0fcff8-48da-4d8b-9147-303c0a477c06 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:57.619025 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:14:57.653826 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0fcff8-48da-4d8b-9147-303c0a477c06 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:14:57.691966 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b359a2b-c0d6-4fb3-9be1-70cfa0a3ae4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:14:57.692568 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b359a2b-c0d6-4fb3-9be1-70cfa0a3ae4f None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:57.692842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b359a2b-c0d6-4fb3-9be1-70cfa0a3ae4f None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:57.693136 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b359a2b-c0d6-4fb3-9be1-70cfa0a3ae4f None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:57.693771 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b359a2b-c0d6-4fb3-9be1-70cfa0a3ae4f None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:14:57.694068 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b359a2b-c0d6-4fb3-9be1-70cfa0a3ae4f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:14:57.695414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b359a2b-c0d6-4fb3-9be1-70cfa0a3ae4f None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:14:57.702382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 592/1166] 10.222.0.85 () {56 vars in 1115 bytes} [Tue Aug 30 04:14:57 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:14:59.572718 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca8a376-fce6-43f9-a61b-02d36278d74f None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:59.573303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca8a376-fce6-43f9-a61b-02d36278d74f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:59.573303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca8a376-fce6-43f9-a61b-02d36278d74f None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:59.574671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 575/1167] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:14:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:14:59.579831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e102f4-5ada-410d-8018-f84ddfa09fda None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:14:59.580232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e102f4-5ada-410d-8018-f84ddfa09fda None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:14:59.580377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e102f4-5ada-410d-8018-f84ddfa09fda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:14:59.599137 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-d9e102f4-5ada-410d-8018-f84ddfa09fda None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:14:59.600305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d9e102f4-5ada-410d-8018-f84ddfa09fda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:14:59.665101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 593/1168] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:14:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:14:59.932630 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca8a376-fce6-43f9-a61b-02d36278d74f None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:14:59.948952 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:00.031977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca8a376-fce6-43f9-a61b-02d36278d74f None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:00.064448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52efb5e7-25d8-4ce2-b7e7-f65e3874235f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:00.065152 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52efb5e7-25d8-4ce2-b7e7-f65e3874235f None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:00.065431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52efb5e7-25d8-4ce2-b7e7-f65e3874235f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:00.065700 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52efb5e7-25d8-4ce2-b7e7-f65e3874235f None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:00.066369 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52efb5e7-25d8-4ce2-b7e7-f65e3874235f None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:15:00.067719 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52efb5e7-25d8-4ce2-b7e7-f65e3874235f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:15:00.083678 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 576/1169] 10.222.0.85 () {60 vars in 1138 bytes} [Tue Aug 30 04:14:59 2022] POST /identity/v3/domains => generated 267 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:15:01.999626 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba4c675-8ffc-4d6b-ab3b-e80262ec4b89 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:02.000180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba4c675-8ffc-4d6b-ab3b-e80262ec4b89 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:02.000180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba4c675-8ffc-4d6b-ab3b-e80262ec4b89 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:02.001549 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 594/1170] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:15:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:15:02.007463 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cadaec-9cdd-4776-8d9f-22974eb5a176 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:02.007912 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cadaec-9cdd-4776-8d9f-22974eb5a176 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:02.008005 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cadaec-9cdd-4776-8d9f-22974eb5a176 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:02.026372 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-a9cadaec-9cdd-4776-8d9f-22974eb5a176 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:15:02.027537 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a9cadaec-9cdd-4776-8d9f-22974eb5a176 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:15:02.091164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 577/1171] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:15:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:15:02.361471 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba4c675-8ffc-4d6b-ab3b-e80262ec4b89 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:02.420541 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:02.461284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba4c675-8ffc-4d6b-ab3b-e80262ec4b89 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:02.494072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578f5195-ef61-4b10-b414-cd7ddc7f4e00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:02.494682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578f5195-ef61-4b10-b414-cd7ddc7f4e00 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:02.494959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578f5195-ef61-4b10-b414-cd7ddc7f4e00 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:02.495229 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578f5195-ef61-4b10-b414-cd7ddc7f4e00 None admin] PATH_INFO: `/v3/domains/1bb60a8bcd2043ec85c20c4287bc7295` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:02.506191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578f5195-ef61-4b10-b414-cd7ddc7f4e00 None admin] RBAC: Authorizing `identity:get_domain(domain_id=1bb60a8bcd2043ec85c20c4287bc7295)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:15:02.507618 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578f5195-ef61-4b10-b414-cd7ddc7f4e00 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:15:02.509187 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 595/1172] 10.222.0.85 () {56 vars in 1183 bytes} [Tue Aug 30 04:15:02 2022] GET /identity/v3/domains/1bb60a8bcd2043ec85c20c4287bc7295 => generated 267 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:15:02.515003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cadaec-9cdd-4776-8d9f-22974eb5a176 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:02.518111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:02.610798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cadaec-9cdd-4776-8d9f-22974eb5a176 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:02.644736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2b6970-30f4-4c67-8772-4b0b77ed8703 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:02.645291 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2b6970-30f4-4c67-8772-4b0b77ed8703 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:02.645568 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2b6970-30f4-4c67-8772-4b0b77ed8703 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:02.645833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2b6970-30f4-4c67-8772-4b0b77ed8703 None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:02.646476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2b6970-30f4-4c67-8772-4b0b77ed8703 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:15:02.647892 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2b6970-30f4-4c67-8772-4b0b77ed8703 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:15:02.674536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 578/1173] 10.222.0.85 () {60 vars in 1141 bytes} [Tue Aug 30 04:15:02 2022] POST /identity/v3/projects => generated 361 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:15:04.608135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07824186-9d7d-44e9-9708-183f84c33c9e None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:04.608673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07824186-9d7d-44e9-9708-183f84c33c9e None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:04.608673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07824186-9d7d-44e9-9708-183f84c33c9e None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:04.610058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 596/1174] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:15:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:15:04.615685 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea62395-8917-4d60-b769-b9774d0bfe37 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:04.616074 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea62395-8917-4d60-b769-b9774d0bfe37 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:04.616232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea62395-8917-4d60-b769-b9774d0bfe37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:04.634934 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-fea62395-8917-4d60-b769-b9774d0bfe37 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:15:04.636117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fea62395-8917-4d60-b769-b9774d0bfe37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:15:04.685136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:04.703092 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 579/1175] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:15:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:15:04.970868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07824186-9d7d-44e9-9708-183f84c33c9e None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:05.071395 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07824186-9d7d-44e9-9708-183f84c33c9e None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:05.091394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:05.105699 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c60c95e-d92a-49eb-9dab-5e95a8595ca2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:05.106339 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c60c95e-d92a-49eb-9dab-5e95a8595ca2 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:05.106621 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c60c95e-d92a-49eb-9dab-5e95a8595ca2 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:05.106895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c60c95e-d92a-49eb-9dab-5e95a8595ca2 None admin] PATH_INFO: `/v3/domains/1bb60a8bcd2043ec85c20c4287bc7295` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:05.108032 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c60c95e-d92a-49eb-9dab-5e95a8595ca2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=1bb60a8bcd2043ec85c20c4287bc7295)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:15:05.109510 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c60c95e-d92a-49eb-9dab-5e95a8595ca2 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:15:05.111086 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 597/1176] 10.222.0.85 () {56 vars in 1183 bytes} [Tue Aug 30 04:15:04 2022] GET /identity/v3/domains/1bb60a8bcd2043ec85c20c4287bc7295 => generated 267 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:15:05.117061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea62395-8917-4d60-b769-b9774d0bfe37 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:05.213092 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea62395-8917-4d60-b769-b9774d0bfe37 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:05.246621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70123527-e0c6-4fa5-910f-681d9ec22990 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:05.247155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70123527-e0c6-4fa5-910f-681d9ec22990 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:05.247427 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70123527-e0c6-4fa5-910f-681d9ec22990 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:05.247694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70123527-e0c6-4fa5-910f-681d9ec22990 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:05.248330 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70123527-e0c6-4fa5-910f-681d9ec22990 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:15:05.249768 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70123527-e0c6-4fa5-910f-681d9ec22990 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:15:05.271605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 580/1177] 10.222.0.85 () {60 vars in 1132 bytes} [Tue Aug 30 04:15:05 2022] POST /identity/v3/users => generated 311 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:15:07.191569 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5e346b-4fb2-4904-bb82-abf28d4c9057 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:07.192080 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5e346b-4fb2-4904-bb82-abf28d4c9057 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:07.192188 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5e346b-4fb2-4904-bb82-abf28d4c9057 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:07.193490 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 598/1178] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:15:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:15:07.198621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8f710b-045f-4171-984f-abdeb5ff17b9 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:07.199061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8f710b-045f-4171-984f-abdeb5ff17b9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:07.199193 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8f710b-045f-4171-984f-abdeb5ff17b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:07.217281 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-6e8f710b-045f-4171-984f-abdeb5ff17b9 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:15:07.218468 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6e8f710b-045f-4171-984f-abdeb5ff17b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:15:07.252834 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:07.283216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 581/1179] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:15:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:15:07.551477 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5e346b-4fb2-4904-bb82-abf28d4c9057 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:07.649875 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5e346b-4fb2-4904-bb82-abf28d4c9057 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:07.683347 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a152949-d153-4a88-bf8b-b8a164aba3c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:07.683965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a152949-d153-4a88-bf8b-b8a164aba3c9 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:07.684237 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a152949-d153-4a88-bf8b-b8a164aba3c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:07.684510 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a152949-d153-4a88-bf8b-b8a164aba3c9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:07.695067 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a152949-d153-4a88-bf8b-b8a164aba3c9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:15:07.699829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-3a152949-d153-4a88-bf8b-b8a164aba3c9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:15:07.699829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:15:07.699829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:15:07.699829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:15:07.699829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:15:07.699829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:15:07.699829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:15:07.699829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:15:07.699829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:15:07.699829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:15:07.699829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:15:07.699829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:15:07.699829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:15:07.699829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:15:07.699829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:15:07.699829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:15:07.700957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:15:07.700957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:15:07.700957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:15:07.700957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:15:07.700957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:15:07.700957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:15:07.700957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:15:07.700957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:15:07.700957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:15:07.700957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:15:07.700957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:15:07.700957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:15:07.700957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:15:07.700957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:15:07.700957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:15:07.700957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:15:07.702072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:15:07.702072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:15:07.702072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:15:07.702072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:15:07.702072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:15:07.702072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:15:07.702072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:15:07.702072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:15:07.702072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:15:07.702072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:15:07.702072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:15:07.702072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:15:07.702072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 599/1180] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:15:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:15:07.707674 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8f710b-045f-4171-984f-abdeb5ff17b9 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:07.805668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8f710b-045f-4171-984f-abdeb5ff17b9 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:07.837951 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfa6966-20cf-4627-85f6-61277f9387b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:07.838548 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfa6966-20cf-4627-85f6-61277f9387b7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:07.838824 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfa6966-20cf-4627-85f6-61277f9387b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:07.839093 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfa6966-20cf-4627-85f6-61277f9387b7 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:07.839756 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfa6966-20cf-4627-85f6-61277f9387b7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:15:07.840055 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfa6966-20cf-4627-85f6-61277f9387b7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:15:07.841446 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfa6966-20cf-4627-85f6-61277f9387b7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:15:07.849734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 582/1181] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:15:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:15:07.855630 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a152949-d153-4a88-bf8b-b8a164aba3c9 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:07.904186 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:07.952780 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a152949-d153-4a88-bf8b-b8a164aba3c9 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:07.984592 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a265a0-7d1f-4087-a94c-d710389badc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:07.985220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a265a0-7d1f-4087-a94c-d710389badc9 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:07.985495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a265a0-7d1f-4087-a94c-d710389badc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:07.985762 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a265a0-7d1f-4087-a94c-d710389badc9 None admin] PATH_INFO: `/v3/users/4d89de40a98c4877b4d78445dfdbc4da` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:07.998304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a265a0-7d1f-4087-a94c-d710389badc9 None admin] RBAC: Authorizing `identity:get_user(user_id=4d89de40a98c4877b4d78445dfdbc4da)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:15:07.999726 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a265a0-7d1f-4087-a94c-d710389badc9 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:15:08.001317 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 600/1182] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:15:07 2022] GET /identity/v3/users/4d89de40a98c4877b4d78445dfdbc4da => generated 311 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:15:08.007990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfa6966-20cf-4627-85f6-61277f9387b7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:08.041445 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:08.105120 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfa6966-20cf-4627-85f6-61277f9387b7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:08.138230 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2e1b98-9d60-427a-80ec-6148d647d253 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:08.138833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2e1b98-9d60-427a-80ec-6148d647d253 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:08.139109 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2e1b98-9d60-427a-80ec-6148d647d253 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:08.139379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2e1b98-9d60-427a-80ec-6148d647d253 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:08.204935 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2e1b98-9d60-427a-80ec-6148d647d253 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:15:08.206452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2e1b98-9d60-427a-80ec-6148d647d253 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:15:08.237813 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:08.301788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 583/1183] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:15:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 04:15:08.307988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a265a0-7d1f-4087-a94c-d710389badc9 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:08.403231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a265a0-7d1f-4087-a94c-d710389badc9 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:08.410665 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:08.437093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfe6d1d-c025-4e7e-9168-19c7088565e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:08.437827 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfe6d1d-c025-4e7e-9168-19c7088565e5 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:08.438121 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfe6d1d-c025-4e7e-9168-19c7088565e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:08.438416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfe6d1d-c025-4e7e-9168-19c7088565e5 None admin] PATH_INFO: `/v3/projects/f4381be74e15465e966440fb5cda2aa6` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:08.439554 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdfe6d1d-c025-4e7e-9168-19c7088565e5 None admin] RBAC: Authorizing `identity:get_project(project_id=f4381be74e15465e966440fb5cda2aa6)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:15:08.440988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdfe6d1d-c025-4e7e-9168-19c7088565e5 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:15:08.443177 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 601/1184] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:15:08 2022] GET /identity/v3/projects/f4381be74e15465e966440fb5cda2aa6 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:15:08.449435 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2e1b98-9d60-427a-80ec-6148d647d253 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:08.544651 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2e1b98-9d60-427a-80ec-6148d647d253 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:08.577505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc64701-e832-43eb-912d-ac7cca5f631b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:08.578098 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc64701-e832-43eb-912d-ac7cca5f631b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:08.578374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc64701-e832-43eb-912d-ac7cca5f631b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:08.578650 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc64701-e832-43eb-912d-ac7cca5f631b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:08.579360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc64701-e832-43eb-912d-ac7cca5f631b None admin] RBAC: Adding query filter params (role.id=d17954e7ce544f0d90a512340232c60e, scope.project.id=f4381be74e15465e966440fb5cda2aa6, user.id=4d89de40a98c4877b4d78445dfdbc4da) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:15:08.579667 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc64701-e832-43eb-912d-ac7cca5f631b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:15:08.581110 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc64701-e832-43eb-912d-ac7cca5f631b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:15:08.582823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:08.587305 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 584/1185] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:15:08 2022] GET /identity/v3/role_assignments?user.id=4d89de40a98c4877b4d78445dfdbc4da&scope.project.id=f4381be74e15465e966440fb5cda2aa6&role.id=d17954e7ce544f0d90a512340232c60e => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:15:10.447321 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81137214-c1ab-42bb-b482-f7e056dd8410 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:10.447830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81137214-c1ab-42bb-b482-f7e056dd8410 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:10.447938 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81137214-c1ab-42bb-b482-f7e056dd8410 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:10.449242 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 602/1186] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:15:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:15:10.454383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433c0733-d071-4fa2-b85c-010ddb10ffdf None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:10.454798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433c0733-d071-4fa2-b85c-010ddb10ffdf None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:10.454937 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433c0733-d071-4fa2-b85c-010ddb10ffdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:10.474510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-433c0733-d071-4fa2-b85c-010ddb10ffdf None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:15:10.475695 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-433c0733-d071-4fa2-b85c-010ddb10ffdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:15:10.540412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 585/1187] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:15:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:15:10.806726 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81137214-c1ab-42bb-b482-f7e056dd8410 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:10.906370 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81137214-c1ab-42bb-b482-f7e056dd8410 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:10.938826 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1101acf-9cac-44f3-b5e4-c09a51844c84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:10.939966 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1101acf-9cac-44f3-b5e4-c09a51844c84 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:10.940249 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1101acf-9cac-44f3-b5e4-c09a51844c84 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:10.940519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1101acf-9cac-44f3-b5e4-c09a51844c84 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:10.950314 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1101acf-9cac-44f3-b5e4-c09a51844c84 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:15:10.955735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-a1101acf-9cac-44f3-b5e4-c09a51844c84 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:15:10.955735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:15:10.955735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:15:10.955735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:15:10.955735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:15:10.955735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:15:10.955735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:15:10.955735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:15:10.955735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:15:10.955735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:15:10.955735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:15:10.955735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:15:10.955735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:15:10.955735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:15:10.955735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:15:10.955735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:15:10.956795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:15:10.956795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:15:10.956795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:15:10.956795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:15:10.956795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:15:10.956795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:15:10.956795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:15:10.956795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:15:10.956795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:15:10.956795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:15:10.956795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:15:10.956795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:15:10.956795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:15:10.956795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:15:10.956795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:15:10.956795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:15:10.957973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:15:10.957973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:15:10.957973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:15:10.957973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:15:10.957973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:15:10.957973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:15:10.957973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:15:10.957973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:15:10.957973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:15:10.957973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:15:10.957973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:15:10.957973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:15:10.957973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 603/1188] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:15:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:15:10.962846 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433c0733-d071-4fa2-b85c-010ddb10ffdf None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:11.057214 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433c0733-d071-4fa2-b85c-010ddb10ffdf None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:11.082884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:11.091118 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caad6cd2-08d6-4eb8-8db0-0e67c92bb307 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:11.091726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caad6cd2-08d6-4eb8-8db0-0e67c92bb307 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:11.092003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caad6cd2-08d6-4eb8-8db0-0e67c92bb307 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:11.092277 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caad6cd2-08d6-4eb8-8db0-0e67c92bb307 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:11.092945 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caad6cd2-08d6-4eb8-8db0-0e67c92bb307 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:15:11.093285 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caad6cd2-08d6-4eb8-8db0-0e67c92bb307 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:15:11.094679 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caad6cd2-08d6-4eb8-8db0-0e67c92bb307 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:15:11.103473 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 586/1189] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:15:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:15:11.109299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1101acf-9cac-44f3-b5e4-c09a51844c84 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:11.145206 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:11.209031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1101acf-9cac-44f3-b5e4-c09a51844c84 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:11.242317 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d51f0a1-19cd-4e20-bd2e-aaa73bc7b451 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:11.242940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d51f0a1-19cd-4e20-bd2e-aaa73bc7b451 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:11.243216 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d51f0a1-19cd-4e20-bd2e-aaa73bc7b451 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:11.243486 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d51f0a1-19cd-4e20-bd2e-aaa73bc7b451 None admin] PATH_INFO: `/v3/users/4d89de40a98c4877b4d78445dfdbc4da` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:11.244588 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d51f0a1-19cd-4e20-bd2e-aaa73bc7b451 None admin] RBAC: Authorizing `identity:get_user(user_id=4d89de40a98c4877b4d78445dfdbc4da)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:15:11.246047 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d51f0a1-19cd-4e20-bd2e-aaa73bc7b451 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:15:11.247620 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 604/1190] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:15:11 2022] GET /identity/v3/users/4d89de40a98c4877b4d78445dfdbc4da => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:15:11.253421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caad6cd2-08d6-4eb8-8db0-0e67c92bb307 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:11.349916 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caad6cd2-08d6-4eb8-8db0-0e67c92bb307 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:11.383705 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de178a9-5233-4ce1-965d-2f2f1decd7bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:11.384331 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de178a9-5233-4ce1-965d-2f2f1decd7bd None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:11.384619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de178a9-5233-4ce1-965d-2f2f1decd7bd None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:11.384893 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de178a9-5233-4ce1-965d-2f2f1decd7bd None admin] PATH_INFO: `/v3/projects/f4381be74e15465e966440fb5cda2aa6` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:11.386037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de178a9-5233-4ce1-965d-2f2f1decd7bd None admin] RBAC: Authorizing `identity:get_project(project_id=f4381be74e15465e966440fb5cda2aa6)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:15:11.387482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de178a9-5233-4ce1-965d-2f2f1decd7bd None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:15:11.389132 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 587/1191] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:15:11 2022] GET /identity/v3/projects/f4381be74e15465e966440fb5cda2aa6 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:15:11.395706 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d51f0a1-19cd-4e20-bd2e-aaa73bc7b451 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:11.492261 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d51f0a1-19cd-4e20-bd2e-aaa73bc7b451 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:11.493775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:11.527258 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c312a53c-b9b3-45f3-8d9d-30ef3e4327cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:11.527901 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c312a53c-b9b3-45f3-8d9d-30ef3e4327cb None admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:11.528180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c312a53c-b9b3-45f3-8d9d-30ef3e4327cb None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:11.528454 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c312a53c-b9b3-45f3-8d9d-30ef3e4327cb None admin] PATH_INFO: `/v3/projects/f4381be74e15465e966440fb5cda2aa6/users/4d89de40a98c4877b4d78445dfdbc4da/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:11.530377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c312a53c-b9b3-45f3-8d9d-30ef3e4327cb None admin] RBAC: Authorizing `identity:create_grant(project_id=f4381be74e15465e966440fb5cda2aa6, user_id=4d89de40a98c4877b4d78445dfdbc4da, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:15:11.531805 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c312a53c-b9b3-45f3-8d9d-30ef3e4327cb None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:15:11.543841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 605/1192] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:15:11 2022] PUT /identity/v3/projects/f4381be74e15465e966440fb5cda2aa6/users/4d89de40a98c4877b4d78445dfdbc4da/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:15:13.402747 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f691ccc-752c-438b-ad6d-57d375c08c61 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:13.403265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f691ccc-752c-438b-ad6d-57d375c08c61 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:13.403265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f691ccc-752c-438b-ad6d-57d375c08c61 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:13.404544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 588/1193] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:15:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:15:13.409682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7083ef0-23bb-4312-9297-86304c6f8f29 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:13.410077 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7083ef0-23bb-4312-9297-86304c6f8f29 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:13.410235 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7083ef0-23bb-4312-9297-86304c6f8f29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:13.428532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-a7083ef0-23bb-4312-9297-86304c6f8f29 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:15:13.429751 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-a7083ef0-23bb-4312-9297-86304c6f8f29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:15:13.495049 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 606/1194] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:15:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:15:13.764274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f691ccc-752c-438b-ad6d-57d375c08c61 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:13.823468 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:13.865878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f691ccc-752c-438b-ad6d-57d375c08c61 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:13.899004 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa797fa-23f0-48b8-a60a-3dda7b911b4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:13.899566 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa797fa-23f0-48b8-a60a-3dda7b911b4a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:13.899845 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa797fa-23f0-48b8-a60a-3dda7b911b4a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:13.900118 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa797fa-23f0-48b8-a60a-3dda7b911b4a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:13.910741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa797fa-23f0-48b8-a60a-3dda7b911b4a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:15:13.915501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-faa797fa-23f0-48b8-a60a-3dda7b911b4a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:15:13.915501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:15:13.915501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:15:13.915501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:15:13.915501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:15:13.915501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:15:13.915501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:15:13.915501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:15:13.915501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:15:13.915501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:15:13.915501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:15:13.915501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:15:13.915501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:15:13.915501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:15:13.915501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:15:13.915501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:15:13.916564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:15:13.916564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:15:13.916564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:15:13.916564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:15:13.916564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:15:13.916564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:15:13.916564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:15:13.916564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:15:13.916564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:15:13.916564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:15:13.916564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:15:13.916564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:15:13.916564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:15:13.916564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:15:13.916564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:15:13.916564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:15:13.917637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:15:13.917637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:15:13.917637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:15:13.917637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:15:13.917637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:15:13.917637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:15:13.917637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:15:13.917637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:15:13.917637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:15:13.917637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:15:13.917637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:15:13.917637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:15:13.917637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 589/1195] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:15:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:15:13.922784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7083ef0-23bb-4312-9297-86304c6f8f29 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:14.012013 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:14.022433 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7083ef0-23bb-4312-9297-86304c6f8f29 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:14.056097 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fe9124-c974-470d-8850-53e910501dee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:14.056698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fe9124-c974-470d-8850-53e910501dee None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:14.056978 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fe9124-c974-470d-8850-53e910501dee None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:14.057278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fe9124-c974-470d-8850-53e910501dee None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:14.057945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24fe9124-c974-470d-8850-53e910501dee None admin] RBAC: Adding query filter params (name=admin) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:15:14.058250 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24fe9124-c974-470d-8850-53e910501dee None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:15:14.059617 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24fe9124-c974-470d-8850-53e910501dee None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:15:14.067457 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 607/1196] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:15:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:15:14.073398 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa797fa-23f0-48b8-a60a-3dda7b911b4a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:14.170922 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa797fa-23f0-48b8-a60a-3dda7b911b4a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:14.187884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:14.204659 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff67ba6b-3e9b-475a-ac38-65972478250a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:14.205361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff67ba6b-3e9b-475a-ac38-65972478250a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:14.205628 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff67ba6b-3e9b-475a-ac38-65972478250a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:14.205895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff67ba6b-3e9b-475a-ac38-65972478250a None admin] PATH_INFO: `/v3/users/4d89de40a98c4877b4d78445dfdbc4da` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:14.207060 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff67ba6b-3e9b-475a-ac38-65972478250a None admin] RBAC: Authorizing `identity:get_user(user_id=4d89de40a98c4877b4d78445dfdbc4da)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:15:14.208475 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff67ba6b-3e9b-475a-ac38-65972478250a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:15:14.210073 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 590/1197] 10.222.0.85 () {56 vars in 1177 bytes} [Tue Aug 30 04:15:14 2022] GET /identity/v3/users/4d89de40a98c4877b4d78445dfdbc4da => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:15:14.215993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fe9124-c974-470d-8850-53e910501dee None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:14.312222 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fe9124-c974-470d-8850-53e910501dee None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:14.344953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074b8f80-89c7-46e0-9d19-e338a992ec95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:14.345568 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074b8f80-89c7-46e0-9d19-e338a992ec95 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:14.346464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074b8f80-89c7-46e0-9d19-e338a992ec95 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:14.346737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074b8f80-89c7-46e0-9d19-e338a992ec95 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:14.369566 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:14.412653 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074b8f80-89c7-46e0-9d19-e338a992ec95 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:15:14.414130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074b8f80-89c7-46e0-9d19-e338a992ec95 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:15:14.510834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 608/1198] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:15:14 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 04:15:14.516791 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff67ba6b-3e9b-475a-ac38-65972478250a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:14.613460 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff67ba6b-3e9b-475a-ac38-65972478250a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:14.646491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa6d1af-4424-45ed-9b75-44817ad6cf98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:14.647101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa6d1af-4424-45ed-9b75-44817ad6cf98 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:14.647377 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa6d1af-4424-45ed-9b75-44817ad6cf98 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:14.647646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa6d1af-4424-45ed-9b75-44817ad6cf98 None admin] PATH_INFO: `/v3/projects/f4381be74e15465e966440fb5cda2aa6` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:14.648734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa6d1af-4424-45ed-9b75-44817ad6cf98 None admin] RBAC: Authorizing `identity:get_project(project_id=f4381be74e15465e966440fb5cda2aa6)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:15:14.650211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa6d1af-4424-45ed-9b75-44817ad6cf98 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:15:14.651835 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 591/1199] 10.222.0.85 () {56 vars in 1186 bytes} [Tue Aug 30 04:15:14 2022] GET /identity/v3/projects/f4381be74e15465e966440fb5cda2aa6 => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:15:14.657795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074b8f80-89c7-46e0-9d19-e338a992ec95 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:15:14.711207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:14.755775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074b8f80-89c7-46e0-9d19-e338a992ec95 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:15:14.789811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89553e61-e302-48dd-9f9f-5a9eb1a76dd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:15:14.790407 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89553e61-e302-48dd-9f9f-5a9eb1a76dd6 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:15:14.790685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89553e61-e302-48dd-9f9f-5a9eb1a76dd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:15:14.790955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89553e61-e302-48dd-9f9f-5a9eb1a76dd6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:15:14.791667 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89553e61-e302-48dd-9f9f-5a9eb1a76dd6 None admin] RBAC: Adding query filter params (role.id=d17954e7ce544f0d90a512340232c60e, scope.project.id=f4381be74e15465e966440fb5cda2aa6, user.id=4d89de40a98c4877b4d78445dfdbc4da) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:15:14.791970 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89553e61-e302-48dd-9f9f-5a9eb1a76dd6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:15:14.793407 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89553e61-e302-48dd-9f9f-5a9eb1a76dd6 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:15:14.799512 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 609/1200] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:15:14 2022] GET /identity/v3/role_assignments?user.id=4d89de40a98c4877b4d78445dfdbc4da&scope.project.id=f4381be74e15465e966440fb5cda2aa6&role.id=d17954e7ce544f0d90a512340232c60e => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:15:24.641862 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=73 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:15:24.796596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=41 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:10.886496 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993e1e9f-c70c-4eed-93b2-02b3762314f0 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:10.886496 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993e1e9f-c70c-4eed-93b2-02b3762314f0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:10.886496 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993e1e9f-c70c-4eed-93b2-02b3762314f0 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:10.887906 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 592/1201] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:16:10 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:16:10.895730 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33132846-d13e-461c-9e0a-a553cdfb0a02 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:10.896301 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33132846-d13e-461c-9e0a-a553cdfb0a02 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:10.896704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33132846-d13e-461c-9e0a-a553cdfb0a02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:10.918869 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-33132846-d13e-461c-9e0a-a553cdfb0a02 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:16:10.920350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-33132846-d13e-461c-9e0a-a553cdfb0a02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:16:11.047184 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 610/1202] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:16:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:16:11.376150 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993e1e9f-c70c-4eed-93b2-02b3762314f0 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:11.545990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993e1e9f-c70c-4eed-93b2-02b3762314f0 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:11.598763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5f0195-91dc-4aa1-8299-20999721f1ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:11.599851 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f0195-91dc-4aa1-8299-20999721f1ac None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:11.600438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f0195-91dc-4aa1-8299-20999721f1ac None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:11.600772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f0195-91dc-4aa1-8299-20999721f1ac None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:11.616524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5f0195-91dc-4aa1-8299-20999721f1ac None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:11.618198 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5f0195-91dc-4aa1-8299-20999721f1ac None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:11.640521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-7e5f0195-91dc-4aa1-8299-20999721f1ac None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:11.640521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:11.640521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:11.640521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:11.640521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:11.640521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:11.640521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:11.640521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:11.640521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:11.640521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:11.640521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:11.640521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:11.640521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:16:11.640521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:16:11.640521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:16:11.640521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:16:11.642569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:11.642569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:11.642569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:11.642569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:11.642569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:11.642569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:11.642569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:11.642569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:11.642569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:11.642569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:11.642569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:11.642569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:11.642569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:11.642569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:11.642569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:11.642569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:11.642569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:16:11.643798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:16:11.643798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:11.643798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:16:11.643798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 593/1203] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:16:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:11.648148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33132846-d13e-461c-9e0a-a553cdfb0a02 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:11.839673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33132846-d13e-461c-9e0a-a553cdfb0a02 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:11.888080 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:11.897188 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d10676-415e-4736-bf10-c87bf7986835 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:11.897995 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d10676-415e-4736-bf10-c87bf7986835 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:11.898213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d10676-415e-4736-bf10-c87bf7986835 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:11.898376 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d10676-415e-4736-bf10-c87bf7986835 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:11.899087 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d10676-415e-4736-bf10-c87bf7986835 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:11.899388 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d10676-415e-4736-bf10-c87bf7986835 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:11.900874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d10676-415e-4736-bf10-c87bf7986835 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:11.921309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 611/1204] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:16:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:16:11.923383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5f0195-91dc-4aa1-8299-20999721f1ac None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:12.003615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:12.066475 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5f0195-91dc-4aa1-8299-20999721f1ac None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:12.119059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3596f59c-ce06-4fac-b664-46c3071fdb95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:12.119059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3596f59c-ce06-4fac-b664-46c3071fdb95 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:12.119059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3596f59c-ce06-4fac-b664-46c3071fdb95 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:12.119451 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3596f59c-ce06-4fac-b664-46c3071fdb95 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:12.119781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3596f59c-ce06-4fac-b664-46c3071fdb95 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:12.121295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3596f59c-ce06-4fac-b664-46c3071fdb95 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:12.157180 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 594/1205] 10.222.0.85 () {60 vars in 1132 bytes} [Tue Aug 30 04:16:11 2022] POST /identity/v3/users => generated 251 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:16:14.554254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08d842a-8171-46f7-a325-0adefd703ae4 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:14.554909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08d842a-8171-46f7-a325-0adefd703ae4 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:14.554909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08d842a-8171-46f7-a325-0adefd703ae4 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:14.556495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 612/1206] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:16:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:16:14.562876 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8407cd-ec75-4b55-abe9-22b39bfce56d None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:14.563545 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8407cd-ec75-4b55-abe9-22b39bfce56d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:14.563545 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8407cd-ec75-4b55-abe9-22b39bfce56d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:14.590729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4f8407cd-ec75-4b55-abe9-22b39bfce56d None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:16:14.592330 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4f8407cd-ec75-4b55-abe9-22b39bfce56d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:16:14.671142 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 595/1207] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:16:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:16:15.006072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08d842a-8171-46f7-a325-0adefd703ae4 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:15.142304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08d842a-8171-46f7-a325-0adefd703ae4 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:15.187206 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5cbd02-8c27-4229-a3c6-9f17a053d593 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:15.187878 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5cbd02-8c27-4229-a3c6-9f17a053d593 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:15.188195 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5cbd02-8c27-4229-a3c6-9f17a053d593 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:15.188544 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5cbd02-8c27-4229-a3c6-9f17a053d593 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:15.200956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5cbd02-8c27-4229-a3c6-9f17a053d593 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:16:15.207567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-bf5cbd02-8c27-4229-a3c6-9f17a053d593 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:16:15.207567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:15.207567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:15.207567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:15.207567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:15.207567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:15.207567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:15.207567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:15.207567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:15.207567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:15.207567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:15.207567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:15.207567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:16:15.207567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:16:15.207567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:16:15.207567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:16:15.208633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:16:15.208633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:16:15.208633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:16:15.208633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:16:15.208633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:15.208633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:15.208633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:15.208633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:15.208633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:15.208633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:15.208633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:15.208633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:15.208633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:15.208633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:15.208633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:15.208633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:15.214336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:15.214336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:15.214336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:15.214336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:15.214336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:16:15.214336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:16:15.214336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:16:15.214336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:16:15.214336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:16:15.214336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:16:15.214336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:16:15.214336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:15.214336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 613/1208] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:16:15 2022] GET /identity/v3/roles/service => generated 85 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:15.215743 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8407cd-ec75-4b55-abe9-22b39bfce56d None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:15.281745 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:15.359267 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8407cd-ec75-4b55-abe9-22b39bfce56d None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:15.402577 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32582383-2dc2-4430-9384-e4d74a7a4694 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:15.403202 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32582383-2dc2-4430-9384-e4d74a7a4694 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:15.403481 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32582383-2dc2-4430-9384-e4d74a7a4694 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:15.403751 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32582383-2dc2-4430-9384-e4d74a7a4694 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:15.404428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32582383-2dc2-4430-9384-e4d74a7a4694 None admin] RBAC: Adding query filter params (name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:15.405228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32582383-2dc2-4430-9384-e4d74a7a4694 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:15.406651 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32582383-2dc2-4430-9384-e4d74a7a4694 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:15.416046 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 596/1209] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:16:15 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:15.422327 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5cbd02-8c27-4229-a3c6-9f17a053d593 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:15.488808 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:15.559537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5cbd02-8c27-4229-a3c6-9f17a053d593 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:15.602320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99584fd6-0b6e-4d8a-b0e9-e049909a6415 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:15.603114 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99584fd6-0b6e-4d8a-b0e9-e049909a6415 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:15.603387 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99584fd6-0b6e-4d8a-b0e9-e049909a6415 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:15.603697 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99584fd6-0b6e-4d8a-b0e9-e049909a6415 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:15.622208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99584fd6-0b6e-4d8a-b0e9-e049909a6415 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:15.623677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99584fd6-0b6e-4d8a-b0e9-e049909a6415 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:15.641221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-99584fd6-0b6e-4d8a-b0e9-e049909a6415 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:15.641221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:15.641221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:15.641221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:15.641221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:15.641221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:15.641221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:15.641221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:15.641221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:15.641221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:15.641221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:15.641221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:15.641221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:16:15.641221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:16:15.641221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:16:15.641221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:16:15.642434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:15.642434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:15.642434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:15.642434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:15.642434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:15.642434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:15.642434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:15.642434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:15.642434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:15.642434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:15.642434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:15.642434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:15.642434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:15.642434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:15.642434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:15.642434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:15.642434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:16:15.643360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:16:15.643360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:15.643360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:15.643360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 614/1210] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:16:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:15.648420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32582383-2dc2-4430-9384-e4d74a7a4694 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:15.779808 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:15.782848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32582383-2dc2-4430-9384-e4d74a7a4694 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:15.823688 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca490521-9812-456e-a9a0-d58baf1d8bf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:15.824228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca490521-9812-456e-a9a0-d58baf1d8bf4 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:15.824508 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca490521-9812-456e-a9a0-d58baf1d8bf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:15.824774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca490521-9812-456e-a9a0-d58baf1d8bf4 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:15.825462 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca490521-9812-456e-a9a0-d58baf1d8bf4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:15.825761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca490521-9812-456e-a9a0-d58baf1d8bf4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:15.827171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca490521-9812-456e-a9a0-d58baf1d8bf4 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:15.846049 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 597/1211] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:16:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:15.852490 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99584fd6-0b6e-4d8a-b0e9-e049909a6415 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:15.969982 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99584fd6-0b6e-4d8a-b0e9-e049909a6415 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:15.971466 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:16.010825 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8e1450-cbce-4bde-818d-4fec0d005b49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:16.011464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8e1450-cbce-4bde-818d-4fec0d005b49 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:16.011747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8e1450-cbce-4bde-818d-4fec0d005b49 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:16.012018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8e1450-cbce-4bde-818d-4fec0d005b49 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:16.020250 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8e1450-cbce-4bde-818d-4fec0d005b49 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:16.021744 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8e1450-cbce-4bde-818d-4fec0d005b49 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:16.034857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-8d8e1450-cbce-4bde-818d-4fec0d005b49 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 30 04:16:16.034857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:16.034857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:16.034857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:16.034857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:16.034857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:16.034857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:16.034857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:16.034857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:16.034857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:16.034857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:16.034857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:16.034857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:16:16.034857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:16:16.034857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:16:16.034857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:16:16.037377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:16.037377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:16.037377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:16:16.037377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:16:16.037377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:16:16.037377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:16:16.037377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:16.037377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:16.037377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:16.037377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:16.037377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:16.037377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:16.037377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:16.037377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:16.037377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:16.037377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:16.037377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:16.037377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:16.038349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:16.038349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:16.038349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:16:16.038349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:16:16.038349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:16:16.038349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:16:16.038349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:16:16.038349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:16:16.038349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 30 04:16:16.038349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:16.038349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 615/1212] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:16:15 2022] GET /identity/v3/users/placement => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:16.042554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca490521-9812-456e-a9a0-d58baf1d8bf4 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:16.170645 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca490521-9812-456e-a9a0-d58baf1d8bf4 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:16.214171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b4045a-d59c-4217-97b2-02799fc8a49a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:16.214821 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b4045a-d59c-4217-97b2-02799fc8a49a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:16.215136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b4045a-d59c-4217-97b2-02799fc8a49a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:16.215406 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b4045a-d59c-4217-97b2-02799fc8a49a None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:16.216132 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b4045a-d59c-4217-97b2-02799fc8a49a None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:16.216456 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b4045a-d59c-4217-97b2-02799fc8a49a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:16.218028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b4045a-d59c-4217-97b2-02799fc8a49a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:16.237400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 598/1213] 10.222.0.85 () {56 vars in 1143 bytes} [Tue Aug 30 04:16:16 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:16.243846 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8e1450-cbce-4bde-818d-4fec0d005b49 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:16.369005 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8e1450-cbce-4bde-818d-4fec0d005b49 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:16.411467 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1e10c6-486f-4e10-aeb5-9ad4d11fd69c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:16.412429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1e10c6-486f-4e10-aeb5-9ad4d11fd69c None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:16.412724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1e10c6-486f-4e10-aeb5-9ad4d11fd69c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:16.413100 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1e10c6-486f-4e10-aeb5-9ad4d11fd69c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:16.450312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:16.510437 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1e10c6-486f-4e10-aeb5-9ad4d11fd69c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:16.512125 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1e10c6-486f-4e10-aeb5-9ad4d11fd69c None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:16.657726 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 616/1214] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:16:16 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 04:16:16.664965 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b4045a-d59c-4217-97b2-02799fc8a49a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:16.687262 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:16.801240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b4045a-d59c-4217-97b2-02799fc8a49a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:16.844523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09032884-377a-4ce1-99a6-b5cb055bd967 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:16.845221 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09032884-377a-4ce1-99a6-b5cb055bd967 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:16.845529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09032884-377a-4ce1-99a6-b5cb055bd967 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:16.845816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09032884-377a-4ce1-99a6-b5cb055bd967 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:16.856890 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09032884-377a-4ce1-99a6-b5cb055bd967 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:16.858896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09032884-377a-4ce1-99a6-b5cb055bd967 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:16.878697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-09032884-377a-4ce1-99a6-b5cb055bd967 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:16.878697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:16.878697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:16.878697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:16.878697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:16.878697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:16.878697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:16.878697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:16.878697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:16.878697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:16.878697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:16.878697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:16.878697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:16:16.878697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:16:16.878697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:16:16.878697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:16:16.880159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:16.880159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:16.880159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:16.880159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:16.880159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:16.880159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:16.880159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:16.880159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:16.880159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:16.880159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:16.880159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:16.880159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:16.880159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:16.880159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:16.880159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:16.880159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:16.880159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:16:16.881206 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:16:16.881206 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:16.881206 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:16:16.881206 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 599/1215] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:16:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:16.886858 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1e10c6-486f-4e10-aeb5-9ad4d11fd69c None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:16.965186 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:17.014034 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1e10c6-486f-4e10-aeb5-9ad4d11fd69c None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:17.062643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1d2904-074d-4704-8e85-fa93587a38a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:17.063255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1d2904-074d-4704-8e85-fa93587a38a4 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:17.063597 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1d2904-074d-4704-8e85-fa93587a38a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:17.063861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1d2904-074d-4704-8e85-fa93587a38a4 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:17.064524 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1d2904-074d-4704-8e85-fa93587a38a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:17.064811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1d2904-074d-4704-8e85-fa93587a38a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:17.066383 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1d2904-074d-4704-8e85-fa93587a38a4 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:17.081437 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 617/1216] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:16:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:17.084474 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09032884-377a-4ce1-99a6-b5cb055bd967 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:17.180181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:17.222432 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09032884-377a-4ce1-99a6-b5cb055bd967 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:17.264260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c04ebbf-4ba0-484d-9f51-740711ddc2c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:17.264564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c04ebbf-4ba0-484d-9f51-740711ddc2c8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:17.264777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c04ebbf-4ba0-484d-9f51-740711ddc2c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:17.265029 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c04ebbf-4ba0-484d-9f51-740711ddc2c8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:17.278465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c04ebbf-4ba0-484d-9f51-740711ddc2c8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:17.280553 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c04ebbf-4ba0-484d-9f51-740711ddc2c8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:17.300226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-4c04ebbf-4ba0-484d-9f51-740711ddc2c8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:16:17.300226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:17.300226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:17.300226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:17.300226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:17.300226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:17.300226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:17.300226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:17.300226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:17.300226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:17.300226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:17.300226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:17.300226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:16:17.300226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:16:17.300226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:16:17.302162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:16:17.302162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:17.302162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:17.302162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:17.302162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:17.302162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:17.302162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:17.302162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:17.302162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:17.302162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:17.302162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:17.302162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:17.302162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:17.302162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:17.302162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:17.302162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:17.302162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:17.303748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:16:17.303748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:16:17.303748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:16:17.303748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:16:17.303748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:16:17.303748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:16:17.303748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:16:17.303748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:16:17.303748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 600/1217] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:16:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:17.308740 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1d2904-074d-4704-8e85-fa93587a38a4 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:17.442882 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1d2904-074d-4704-8e85-fa93587a38a4 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:17.475794 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:17.490556 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a22c4d-e428-4334-978f-c72eeabdc139 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:17.491172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a22c4d-e428-4334-978f-c72eeabdc139 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:17.491448 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a22c4d-e428-4334-978f-c72eeabdc139 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:17.491718 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a22c4d-e428-4334-978f-c72eeabdc139 None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:17.492462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a22c4d-e428-4334-978f-c72eeabdc139 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:17.492772 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a22c4d-e428-4334-978f-c72eeabdc139 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:17.494248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a22c4d-e428-4334-978f-c72eeabdc139 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:17.509518 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 618/1218] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:16:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:17.516681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c04ebbf-4ba0-484d-9f51-740711ddc2c8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:17.682457 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c04ebbf-4ba0-484d-9f51-740711ddc2c8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:17.749424 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:17.757910 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ff3667-26e4-49b3-8690-e2e646a832a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:17.758576 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ff3667-26e4-49b3-8690-e2e646a832a6 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:17.758859 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ff3667-26e4-49b3-8690-e2e646a832a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:17.760301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ff3667-26e4-49b3-8690-e2e646a832a6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:17.760987 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ff3667-26e4-49b3-8690-e2e646a832a6 None admin] RBAC: Adding query filter params (role.id=755eb1bfc0cf445d8f282358f404a320, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=359d79cbfcad49beba2b495066824128) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:17.763417 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ff3667-26e4-49b3-8690-e2e646a832a6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:17.766329 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ff3667-26e4-49b3-8690-e2e646a832a6 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:17.773293 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 601/1219] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:16:17 2022] GET /identity/v3/role_assignments?user.id=359d79cbfcad49beba2b495066824128&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=755eb1bfc0cf445d8f282358f404a320 => generated 259 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:19.825027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b096c0a-c11c-40a9-9b6f-2b801813f823 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:19.825702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b096c0a-c11c-40a9-9b6f-2b801813f823 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:19.825702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b096c0a-c11c-40a9-9b6f-2b801813f823 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:19.826938 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 619/1220] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:16:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:16:19.832534 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a580382-80c9-44a7-9f13-5efdb6708e55 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:19.833009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a580382-80c9-44a7-9f13-5efdb6708e55 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:19.833202 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a580382-80c9-44a7-9f13-5efdb6708e55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:19.852978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-9a580382-80c9-44a7-9f13-5efdb6708e55 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:16:19.854186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9a580382-80c9-44a7-9f13-5efdb6708e55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:16:19.928631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 602/1221] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:16:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:16:20.200034 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b096c0a-c11c-40a9-9b6f-2b801813f823 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:20.312610 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b096c0a-c11c-40a9-9b6f-2b801813f823 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:20.348239 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768b0210-b357-4eaa-b83b-f0a119a92f25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:20.348862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768b0210-b357-4eaa-b83b-f0a119a92f25 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:20.349176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768b0210-b357-4eaa-b83b-f0a119a92f25 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:20.349446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768b0210-b357-4eaa-b83b-f0a119a92f25 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:20.360296 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768b0210-b357-4eaa-b83b-f0a119a92f25 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:16:20.365551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-768b0210-b357-4eaa-b83b-f0a119a92f25 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:16:20.365551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:20.365551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:20.365551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:20.365551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:20.365551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:20.365551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:20.365551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:20.365551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:20.365551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:20.365551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:20.365551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:20.365551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:16:20.365551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:16:20.365551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:16:20.365551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:16:20.366698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:16:20.366698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:16:20.366698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:16:20.366698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:16:20.366698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:20.366698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:20.366698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:20.366698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:20.366698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:20.366698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:20.366698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:20.366698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:20.366698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:20.366698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:20.366698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:20.366698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:20.367831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:20.367831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:20.367831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:20.367831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:20.367831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:16:20.367831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:16:20.367831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:16:20.367831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:16:20.367831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:16:20.367831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:16:20.367831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:16:20.367831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:20.367831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 620/1222] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:16:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:16:20.372997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a580382-80c9-44a7-9f13-5efdb6708e55 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:20.477887 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a580382-80c9-44a7-9f13-5efdb6708e55 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:20.502375 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:20.515184 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f11967-8dc1-44e1-9614-0a7e66764d2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:20.516533 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f11967-8dc1-44e1-9614-0a7e66764d2e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:20.516936 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f11967-8dc1-44e1-9614-0a7e66764d2e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:20.517356 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f11967-8dc1-44e1-9614-0a7e66764d2e None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:20.518241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f11967-8dc1-44e1-9614-0a7e66764d2e None admin] RBAC: Adding query filter params (name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:20.518627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f11967-8dc1-44e1-9614-0a7e66764d2e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:20.520467 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f11967-8dc1-44e1-9614-0a7e66764d2e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:20.529325 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 603/1223] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:16:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:20.535717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768b0210-b357-4eaa-b83b-f0a119a92f25 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:20.587803 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:20.652610 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768b0210-b357-4eaa-b83b-f0a119a92f25 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:20.692770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c7d7e8-95ec-4c10-bc20-caefe278a971 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:20.693536 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c7d7e8-95ec-4c10-bc20-caefe278a971 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:20.693842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c7d7e8-95ec-4c10-bc20-caefe278a971 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:20.694141 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c7d7e8-95ec-4c10-bc20-caefe278a971 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:20.705686 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c7d7e8-95ec-4c10-bc20-caefe278a971 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:20.707250 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c7d7e8-95ec-4c10-bc20-caefe278a971 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:20.723304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-a7c7d7e8-95ec-4c10-bc20-caefe278a971 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:20.723304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:20.723304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:20.723304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:20.723304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:20.723304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:20.723304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:20.723304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:20.723304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:20.723304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:20.723304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:20.723304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:20.723304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:16:20.723304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:16:20.723304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:16:20.723304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:16:20.724330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:20.724330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:20.724330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:20.724330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:20.724330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:20.724330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:20.724330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:20.724330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:20.724330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:20.724330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:20.724330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:20.724330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:20.724330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:20.724330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:20.724330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:20.724330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:20.724330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:16:20.725248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:16:20.725248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:20.725248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:20.725248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 621/1224] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:16:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:20.730511 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f11967-8dc1-44e1-9614-0a7e66764d2e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:20.834878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f11967-8dc1-44e1-9614-0a7e66764d2e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:20.869933 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c34ce91-e737-4641-9af3-7571dca899ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:20.870628 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c34ce91-e737-4641-9af3-7571dca899ab None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:20.870816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c34ce91-e737-4641-9af3-7571dca899ab None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:20.871084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c34ce91-e737-4641-9af3-7571dca899ab None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:20.871830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c34ce91-e737-4641-9af3-7571dca899ab None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:20.872128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c34ce91-e737-4641-9af3-7571dca899ab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:20.873523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c34ce91-e737-4641-9af3-7571dca899ab None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:20.885875 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 604/1225] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:16:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:20.892265 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c7d7e8-95ec-4c10-bc20-caefe278a971 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:20.988712 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:20.996349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c7d7e8-95ec-4c10-bc20-caefe278a971 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:21.032469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60b4ab7-cad1-4742-8d4b-c496760f6733 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:21.033065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60b4ab7-cad1-4742-8d4b-c496760f6733 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:21.033449 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60b4ab7-cad1-4742-8d4b-c496760f6733 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:21.033621 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60b4ab7-cad1-4742-8d4b-c496760f6733 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:21.041910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60b4ab7-cad1-4742-8d4b-c496760f6733 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:21.043364 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60b4ab7-cad1-4742-8d4b-c496760f6733 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:21.054845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-a60b4ab7-cad1-4742-8d4b-c496760f6733 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 30 04:16:21.054845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:21.054845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:21.054845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:21.054845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:21.054845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:21.054845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:21.054845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:21.054845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:21.054845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:21.054845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:21.054845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:21.054845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:16:21.054845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:16:21.054845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:16:21.054845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:16:21.056016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:21.056016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:21.056016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:16:21.056016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:16:21.056016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:16:21.056016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:16:21.056016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:21.056016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:21.056016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:21.056016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:21.056016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:21.056016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:21.056016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:21.056016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:21.056016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:21.056016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:21.056016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:21.056016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:21.056972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:21.056972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:21.056972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:16:21.056972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:16:21.056972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:16:21.056972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:16:21.056972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:16:21.056972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:16:21.056972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 30 04:16:21.056972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:21.056972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 622/1226] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:16:20 2022] GET /identity/v3/users/placement => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:21.061983 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c34ce91-e737-4641-9af3-7571dca899ab None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:21.111639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:21.170048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c34ce91-e737-4641-9af3-7571dca899ab None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:21.206420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660bae1c-61be-42b5-8eeb-9df50dd523ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:21.207027 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660bae1c-61be-42b5-8eeb-9df50dd523ae None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:21.207302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660bae1c-61be-42b5-8eeb-9df50dd523ae None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:21.207570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660bae1c-61be-42b5-8eeb-9df50dd523ae None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:21.208292 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660bae1c-61be-42b5-8eeb-9df50dd523ae None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:21.208591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660bae1c-61be-42b5-8eeb-9df50dd523ae None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:21.210040 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660bae1c-61be-42b5-8eeb-9df50dd523ae None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:21.235375 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 605/1227] 10.222.0.85 () {56 vars in 1143 bytes} [Tue Aug 30 04:16:21 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:16:21.236824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60b4ab7-cad1-4742-8d4b-c496760f6733 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:21.348727 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60b4ab7-cad1-4742-8d4b-c496760f6733 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:21.387638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a76855-d6b6-4c2d-a3aa-059e9f27116e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:21.388279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a76855-d6b6-4c2d-a3aa-059e9f27116e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:21.388553 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a76855-d6b6-4c2d-a3aa-059e9f27116e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:21.388827 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a76855-d6b6-4c2d-a3aa-059e9f27116e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:21.400590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a76855-d6b6-4c2d-a3aa-059e9f27116e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:21.402151 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a76855-d6b6-4c2d-a3aa-059e9f27116e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:21.418279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-90a76855-d6b6-4c2d-a3aa-059e9f27116e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:21.418279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:21.418279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:21.418279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:21.418279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:21.418279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:21.418279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:21.418279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:21.418279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:21.418279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:21.418279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:21.418279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:21.418279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:16:21.418279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:16:21.418279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:16:21.418279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:16:21.424790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:21.424790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:21.424790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:21.424790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:21.424790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:21.424790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:21.424790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:21.424790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:21.424790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:21.424790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:21.424790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:21.424790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:21.424790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:21.424790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:21.424790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:21.424790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:21.424790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:16:21.425940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:16:21.425940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:21.425940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:21.425940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 623/1228] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:16:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:21.426230 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660bae1c-61be-42b5-8eeb-9df50dd523ae None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:21.537474 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660bae1c-61be-42b5-8eeb-9df50dd523ae None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:21.540232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:21.577012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997d011c-8517-4c41-8749-c31d180ce0e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:21.577719 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997d011c-8517-4c41-8749-c31d180ce0e2 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:21.578047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997d011c-8517-4c41-8749-c31d180ce0e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:21.578317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997d011c-8517-4c41-8749-c31d180ce0e2 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:21.578913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997d011c-8517-4c41-8749-c31d180ce0e2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:21.579207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997d011c-8517-4c41-8749-c31d180ce0e2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:21.580621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997d011c-8517-4c41-8749-c31d180ce0e2 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:21.595318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 606/1229] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:16:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:16:21.602008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a76855-d6b6-4c2d-a3aa-059e9f27116e None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:21.612115 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:21.717169 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a76855-d6b6-4c2d-a3aa-059e9f27116e None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:21.757607 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18671caa-c1ac-4639-a7f8-9649baeb2983 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:21.758238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18671caa-c1ac-4639-a7f8-9649baeb2983 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:21.758512 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18671caa-c1ac-4639-a7f8-9649baeb2983 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:21.758782 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18671caa-c1ac-4639-a7f8-9649baeb2983 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:21.765650 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18671caa-c1ac-4639-a7f8-9649baeb2983 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:21.767089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18671caa-c1ac-4639-a7f8-9649baeb2983 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:21.778118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-18671caa-c1ac-4639-a7f8-9649baeb2983 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:16:21.778118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:21.778118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:21.778118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:21.778118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:21.778118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:21.778118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:21.778118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:21.778118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:21.778118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:21.778118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:21.778118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:21.778118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:16:21.778118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:16:21.778118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:16:21.779270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:16:21.779270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:21.779270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:21.779270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:21.779270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:21.779270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:21.779270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:21.779270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:21.779270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:21.779270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:21.779270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:21.779270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:21.779270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:21.779270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:21.779270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:21.779270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:21.779270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:21.780599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:16:21.780599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:16:21.780599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:16:21.780599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:16:21.780599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:16:21.780599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:16:21.780599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:16:21.780599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:21.780599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 624/1230] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:16:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:16:21.785392 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997d011c-8517-4c41-8749-c31d180ce0e2 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:21.892489 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997d011c-8517-4c41-8749-c31d180ce0e2 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:21.943971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f659c7-7d56-47c7-9cdd-f57d3f0c5ed8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:21.944683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f659c7-7d56-47c7-9cdd-f57d3f0c5ed8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:21.944987 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f659c7-7d56-47c7-9cdd-f57d3f0c5ed8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:21.945453 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f659c7-7d56-47c7-9cdd-f57d3f0c5ed8 None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:21.946062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f659c7-7d56-47c7-9cdd-f57d3f0c5ed8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:21.946337 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f659c7-7d56-47c7-9cdd-f57d3f0c5ed8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:21.947768 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f659c7-7d56-47c7-9cdd-f57d3f0c5ed8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:21.966358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 607/1231] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:16:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:21.974177 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18671caa-c1ac-4639-a7f8-9649baeb2983 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:22.051113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:22.099732 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18671caa-c1ac-4639-a7f8-9649baeb2983 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:22.145312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa28d00-e169-4a18-b1b8-fd4aee636a4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:22.145897 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa28d00-e169-4a18-b1b8-fd4aee636a4c None admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:22.146174 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa28d00-e169-4a18-b1b8-fd4aee636a4c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:22.146487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa28d00-e169-4a18-b1b8-fd4aee636a4c None admin] PATH_INFO: `/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/359d79cbfcad49beba2b495066824128/roles/755eb1bfc0cf445d8f282358f404a320` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:22.163740 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa28d00-e169-4a18-b1b8-fd4aee636a4c None admin] RBAC: Authorizing `identity:create_grant(project_id=a860dcc3b2d64d1491c40710ec7423a5, user_id=359d79cbfcad49beba2b495066824128, role_id=755eb1bfc0cf445d8f282358f404a320)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:22.165489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa28d00-e169-4a18-b1b8-fd4aee636a4c None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:22.181031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 625/1232] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:16:21 2022] PUT /identity/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/359d79cbfcad49beba2b495066824128/roles/755eb1bfc0cf445d8f282358f404a320 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:16:24.149409 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65792a5-5955-4416-bbef-67292039277f None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:24.150011 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65792a5-5955-4416-bbef-67292039277f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:24.150011 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65792a5-5955-4416-bbef-67292039277f None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:24.151299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 608/1233] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:16:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:16:24.156531 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59299060-ad7c-4b89-ab25-68cbcd659be6 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:24.156950 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59299060-ad7c-4b89-ab25-68cbcd659be6 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:24.157078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59299060-ad7c-4b89-ab25-68cbcd659be6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:24.175506 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-59299060-ad7c-4b89-ab25-68cbcd659be6 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:16:24.176768 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-59299060-ad7c-4b89-ab25-68cbcd659be6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:16:24.248466 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 626/1234] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:16:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:16:24.519749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65792a5-5955-4416-bbef-67292039277f None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:24.551539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:24.637702 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65792a5-5955-4416-bbef-67292039277f None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:24.677435 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081bf2a8-8aaf-4c9c-8bdd-6e5559219392 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:24.678471 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081bf2a8-8aaf-4c9c-8bdd-6e5559219392 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:24.678817 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081bf2a8-8aaf-4c9c-8bdd-6e5559219392 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:24.679139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081bf2a8-8aaf-4c9c-8bdd-6e5559219392 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:24.695860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081bf2a8-8aaf-4c9c-8bdd-6e5559219392 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:16:24.706787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-081bf2a8-8aaf-4c9c-8bdd-6e5559219392 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:16:24.706787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:24.706787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:24.706787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:24.706787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:24.706787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:24.706787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:24.706787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:24.706787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:24.706787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:24.706787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:24.706787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:24.706787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:16:24.706787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:16:24.706787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:16:24.706787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:16:24.708035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:16:24.708035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:16:24.708035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:16:24.708035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:16:24.708035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:24.708035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:24.708035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:24.708035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:24.708035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:24.708035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:24.708035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:24.708035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:24.708035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:24.708035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:24.708035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:24.708035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:24.708889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:24.708889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:24.708889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:24.708889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:24.708889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:16:24.708889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:16:24.708889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:16:24.708889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:16:24.708889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:16:24.708889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:16:24.708889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 04:16:24.708889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:16:24.710091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 609/1235] 10.222.0.85 () {56 vars in 1102 bytes} [Tue Aug 30 04:16:24 2022] GET /identity/v3/roles/service => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:24.715624 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59299060-ad7c-4b89-ab25-68cbcd659be6 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:24.770958 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:24.841766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59299060-ad7c-4b89-ab25-68cbcd659be6 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:24.885863 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb3dd7f-d80f-4691-b850-7d84cc4cc97e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:24.886765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb3dd7f-d80f-4691-b850-7d84cc4cc97e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:24.887171 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb3dd7f-d80f-4691-b850-7d84cc4cc97e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:24.887563 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb3dd7f-d80f-4691-b850-7d84cc4cc97e None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:24.888623 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb3dd7f-d80f-4691-b850-7d84cc4cc97e None admin] RBAC: Adding query filter params (name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:24.889124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb3dd7f-d80f-4691-b850-7d84cc4cc97e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:24.891920 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb3dd7f-d80f-4691-b850-7d84cc4cc97e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:24.902042 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 627/1236] 10.222.0.85 () {56 vars in 1103 bytes} [Tue Aug 30 04:16:24 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:24.909865 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081bf2a8-8aaf-4c9c-8bdd-6e5559219392 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:25.022773 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:25.040123 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081bf2a8-8aaf-4c9c-8bdd-6e5559219392 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:25.088151 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869265dc-1bac-4109-9493-f84f7aa37fd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:25.088824 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869265dc-1bac-4109-9493-f84f7aa37fd7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:25.089242 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869265dc-1bac-4109-9493-f84f7aa37fd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:25.089493 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869265dc-1bac-4109-9493-f84f7aa37fd7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:25.104289 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869265dc-1bac-4109-9493-f84f7aa37fd7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:25.108200 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869265dc-1bac-4109-9493-f84f7aa37fd7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:25.127570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-869265dc-1bac-4109-9493-f84f7aa37fd7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:25.127570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:25.127570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:25.127570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:25.127570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:25.127570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:25.127570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:25.127570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:25.127570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:25.127570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:25.127570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:25.127570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:25.127570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:16:25.127570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:16:25.127570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:16:25.127570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:16:25.129030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:25.129030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:25.129030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:25.129030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:25.129030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:25.129030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:25.129030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:25.129030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:25.129030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:25.129030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:25.129030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:25.129030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:25.129030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:25.129030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:25.129030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:25.129030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:25.129030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:16:25.129982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:16:25.129982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:25.129982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:16:25.129982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 610/1237] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:16:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:25.135130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb3dd7f-d80f-4691-b850-7d84cc4cc97e None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:25.256993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb3dd7f-d80f-4691-b850-7d84cc4cc97e None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:25.266863 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:25.293774 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40445a4-a3e4-45ee-920c-66e5fb01c525 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:25.294311 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40445a4-a3e4-45ee-920c-66e5fb01c525 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:25.294588 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40445a4-a3e4-45ee-920c-66e5fb01c525 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:25.294857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40445a4-a3e4-45ee-920c-66e5fb01c525 None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:25.295504 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40445a4-a3e4-45ee-920c-66e5fb01c525 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:25.295803 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40445a4-a3e4-45ee-920c-66e5fb01c525 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:25.297216 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40445a4-a3e4-45ee-920c-66e5fb01c525 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:25.309865 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 628/1238] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:16:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:25.316379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869265dc-1bac-4109-9493-f84f7aa37fd7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:25.430111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869265dc-1bac-4109-9493-f84f7aa37fd7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:25.470885 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d0510d-c732-4e3f-8c44-393bd3f0606b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:25.471527 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d0510d-c732-4e3f-8c44-393bd3f0606b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:25.471807 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d0510d-c732-4e3f-8c44-393bd3f0606b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:25.472080 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d0510d-c732-4e3f-8c44-393bd3f0606b None admin] PATH_INFO: `/v3/users/placement` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:25.475269 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:25.481167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d0510d-c732-4e3f-8c44-393bd3f0606b None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:25.482928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d0510d-c732-4e3f-8c44-393bd3f0606b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:25.497194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-33d0510d-c732-4e3f-8c44-393bd3f0606b None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 30 04:16:25.497194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:25.497194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:25.497194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:25.497194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:25.497194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:25.497194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:25.497194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:25.497194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:25.497194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:25.497194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:25.497194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:25.497194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:16:25.497194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:16:25.497194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:16:25.497194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:16:25.498742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:25.498742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:25.498742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:16:25.498742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:16:25.498742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:16:25.498742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:16:25.498742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:25.498742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:25.498742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:25.498742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:25.498742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:25.498742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:25.498742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:25.498742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:25.498742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:25.498742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:25.498742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:25.498742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:25.501003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:25.501003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:25.501003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:16:25.501003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:16:25.501003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:16:25.501003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:16:25.501003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:16:25.501003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:16:25.501003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 30 04:16:25.501003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:16:25.501003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 611/1239] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:16:25 2022] GET /identity/v3/users/placement => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:25.506216 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40445a4-a3e4-45ee-920c-66e5fb01c525 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:25.615756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40445a4-a3e4-45ee-920c-66e5fb01c525 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:25.658462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d22ed36-6327-4cb5-934d-7da12d10aea6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:25.659205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d22ed36-6327-4cb5-934d-7da12d10aea6 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:25.659449 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d22ed36-6327-4cb5-934d-7da12d10aea6 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:25.659791 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d22ed36-6327-4cb5-934d-7da12d10aea6 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:25.660660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d22ed36-6327-4cb5-934d-7da12d10aea6 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:25.661125 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d22ed36-6327-4cb5-934d-7da12d10aea6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:25.662651 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d22ed36-6327-4cb5-934d-7da12d10aea6 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:25.689226 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 629/1240] 10.222.0.85 () {56 vars in 1143 bytes} [Tue Aug 30 04:16:25 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:25.695605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d0510d-c732-4e3f-8c44-393bd3f0606b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:25.817503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d0510d-c732-4e3f-8c44-393bd3f0606b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:25.865177 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7023493b-c43a-4f73-8b5d-0f97090082b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:25.865935 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7023493b-c43a-4f73-8b5d-0f97090082b4 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:25.866157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7023493b-c43a-4f73-8b5d-0f97090082b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:25.866434 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7023493b-c43a-4f73-8b5d-0f97090082b4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:25.944738 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:25.952971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7023493b-c43a-4f73-8b5d-0f97090082b4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:25.954471 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7023493b-c43a-4f73-8b5d-0f97090082b4 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:26.072149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 612/1241] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:16:25 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 04:16:26.078397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d22ed36-6327-4cb5-934d-7da12d10aea6 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:26.123155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:26.223390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d22ed36-6327-4cb5-934d-7da12d10aea6 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:26.265805 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bd26e6-f28c-41a1-a074-d1f10e23723f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:26.266513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bd26e6-f28c-41a1-a074-d1f10e23723f None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:26.266798 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bd26e6-f28c-41a1-a074-d1f10e23723f None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:26.267062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bd26e6-f28c-41a1-a074-d1f10e23723f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:26.281300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bd26e6-f28c-41a1-a074-d1f10e23723f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:26.282797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bd26e6-f28c-41a1-a074-d1f10e23723f None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:26.299832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-f1bd26e6-f28c-41a1-a074-d1f10e23723f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:26.299832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:26.299832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:26.299832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:26.299832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:26.299832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:26.299832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:26.299832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:26.299832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:26.299832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:26.299832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:26.299832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:26.299832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:16:26.299832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:16:26.299832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:16:26.299832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:16:26.301207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:26.301207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:26.301207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:26.301207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:26.301207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:26.301207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:26.301207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:26.301207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:26.301207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:26.301207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:26.301207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:26.301207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:26.301207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:26.301207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:26.301207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:26.301207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:26.301207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:16:26.302155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:16:26.302155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:26.302155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:26.302155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 630/1242] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:16:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:26.307227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7023493b-c43a-4f73-8b5d-0f97090082b4 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:26.453985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:26.460563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7023493b-c43a-4f73-8b5d-0f97090082b4 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:26.503470 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddc0acc-7936-49e1-984b-6b08a98132b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:26.504310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddc0acc-7936-49e1-984b-6b08a98132b9 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:26.504584 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddc0acc-7936-49e1-984b-6b08a98132b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:26.504858 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddc0acc-7936-49e1-984b-6b08a98132b9 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:26.505545 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddc0acc-7936-49e1-984b-6b08a98132b9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:26.505863 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddc0acc-7936-49e1-984b-6b08a98132b9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:26.507259 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddc0acc-7936-49e1-984b-6b08a98132b9 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:26.524671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 613/1243] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:16:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:16:26.525956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bd26e6-f28c-41a1-a074-d1f10e23723f None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:26.631407 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:26.663001 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bd26e6-f28c-41a1-a074-d1f10e23723f None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:26.710465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982c8a89-ca36-42f7-b31e-a77bbca96e13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:26.711456 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982c8a89-ca36-42f7-b31e-a77bbca96e13 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:26.712011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982c8a89-ca36-42f7-b31e-a77bbca96e13 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:26.712452 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982c8a89-ca36-42f7-b31e-a77bbca96e13 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:26.722497 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982c8a89-ca36-42f7-b31e-a77bbca96e13 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:26.724103 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982c8a89-ca36-42f7-b31e-a77bbca96e13 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:26.738747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-982c8a89-ca36-42f7-b31e-a77bbca96e13 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:16:26.738747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:26.738747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:26.738747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:26.738747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:26.738747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:26.738747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:26.738747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:26.738747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:26.738747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:26.738747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:26.738747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:26.738747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:16:26.738747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:16:26.738747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:16:26.740154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:16:26.740154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:26.740154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:26.740154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:26.740154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:26.740154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:26.740154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:26.740154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:26.740154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:26.740154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:26.740154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:26.740154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:26.740154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:26.740154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:26.740154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:26.740154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:26.740154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:26.741224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:16:26.741224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:16:26.741224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:16:26.741224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:16:26.741224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:16:26.741224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:16:26.741224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:16:26.741224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:26.741224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 631/1244] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:16:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:26.749062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddc0acc-7936-49e1-984b-6b08a98132b9 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:26.885424 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddc0acc-7936-49e1-984b-6b08a98132b9 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:26.926266 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1a01bf-51d4-4c00-b8e7-1e2ab60089fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:26.926266 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1a01bf-51d4-4c00-b8e7-1e2ab60089fc None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:26.926266 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1a01bf-51d4-4c00-b8e7-1e2ab60089fc None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:26.926266 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1a01bf-51d4-4c00-b8e7-1e2ab60089fc None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:26.926907 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1a01bf-51d4-4c00-b8e7-1e2ab60089fc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:26.926907 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1a01bf-51d4-4c00-b8e7-1e2ab60089fc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:26.926907 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1a01bf-51d4-4c00-b8e7-1e2ab60089fc None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:26.943113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 614/1245] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:16:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:26.949829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982c8a89-ca36-42f7-b31e-a77bbca96e13 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:27.084342 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982c8a89-ca36-42f7-b31e-a77bbca96e13 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:27.124412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2ab1f3-dd33-4c64-8234-da5dd7218c77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:27.125796 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2ab1f3-dd33-4c64-8234-da5dd7218c77 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:27.126131 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2ab1f3-dd33-4c64-8234-da5dd7218c77 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:27.126573 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2ab1f3-dd33-4c64-8234-da5dd7218c77 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:27.127419 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2ab1f3-dd33-4c64-8234-da5dd7218c77 None admin] RBAC: Adding query filter params (role.id=755eb1bfc0cf445d8f282358f404a320, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=359d79cbfcad49beba2b495066824128) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:27.127737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2ab1f3-dd33-4c64-8234-da5dd7218c77 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:27.129589 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2ab1f3-dd33-4c64-8234-da5dd7218c77 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:27.136664 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 632/1246] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:16:26 2022] GET /identity/v3/role_assignments?user.id=359d79cbfcad49beba2b495066824128&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=755eb1bfc0cf445d8f282358f404a320 => generated 608 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:16:29.296865 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd365d62-345f-419d-a891-950995e5bd9e None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:29.297737 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd365d62-345f-419d-a891-950995e5bd9e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:29.297737 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd365d62-345f-419d-a891-950995e5bd9e None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:29.299043 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 615/1247] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:16:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:16:29.308588 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7304821-aa18-4924-a1f9-47d5102f3067 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:29.309156 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7304821-aa18-4924-a1f9-47d5102f3067 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:29.309446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7304821-aa18-4924-a1f9-47d5102f3067 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:29.316809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:29.331749 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-d7304821-aa18-4924-a1f9-47d5102f3067 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:16:29.333125 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d7304821-aa18-4924-a1f9-47d5102f3067 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:16:29.413504 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 633/1248] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:16:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:16:29.707095 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd365d62-345f-419d-a891-950995e5bd9e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:29.728491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:29.844037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd365d62-345f-419d-a891-950995e5bd9e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:29.892819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f660072-aed5-4b18-a357-24c8e54465d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:29.893574 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f660072-aed5-4b18-a357-24c8e54465d8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:29.893845 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f660072-aed5-4b18-a357-24c8e54465d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:29.894156 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f660072-aed5-4b18-a357-24c8e54465d8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:29.908698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f660072-aed5-4b18-a357-24c8e54465d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:16:29.914338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-6f660072-aed5-4b18-a357-24c8e54465d8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:16:29.914338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:29.914338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:29.914338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:29.914338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:29.914338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:29.914338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:29.914338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:29.914338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:29.914338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:29.914338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:29.914338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:29.914338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:16:29.914338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:16:29.914338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:16:29.914338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:16:29.915589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:16:29.915589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:16:29.915589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:16:29.915589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:16:29.915589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:29.915589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:29.915589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:29.915589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:29.915589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:29.915589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:29.915589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:29.915589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:29.915589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:29.915589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:29.915589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:29.915589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:29.916448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:29.916448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:29.916448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:29.916448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:29.916448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:16:29.916448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:16:29.916448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:16:29.916448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:16:29.916448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:16:29.916448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:16:29.916448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:16:29.916448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:16:29.916448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 616/1249] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:16:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:29.922238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7304821-aa18-4924-a1f9-47d5102f3067 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:30.050402 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7304821-aa18-4924-a1f9-47d5102f3067 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:30.084041 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:30.090138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e477767-a972-4aa3-917c-e9d2d4e8304c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:30.090762 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e477767-a972-4aa3-917c-e9d2d4e8304c None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:30.091046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e477767-a972-4aa3-917c-e9d2d4e8304c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:30.091318 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e477767-a972-4aa3-917c-e9d2d4e8304c None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:30.091995 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e477767-a972-4aa3-917c-e9d2d4e8304c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:30.092298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e477767-a972-4aa3-917c-e9d2d4e8304c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:30.094428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e477767-a972-4aa3-917c-e9d2d4e8304c None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:30.104919 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 634/1250] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:16:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:30.111678 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f660072-aed5-4b18-a357-24c8e54465d8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:30.211376 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:30.240548 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f660072-aed5-4b18-a357-24c8e54465d8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:30.279386 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7375d2-03bc-4215-8200-aee948f366ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:30.280031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7375d2-03bc-4215-8200-aee948f366ba None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:30.280311 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7375d2-03bc-4215-8200-aee948f366ba None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:30.280616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7375d2-03bc-4215-8200-aee948f366ba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:30.293007 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7375d2-03bc-4215-8200-aee948f366ba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:30.294497 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7375d2-03bc-4215-8200-aee948f366ba None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:30.311567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-4c7375d2-03bc-4215-8200-aee948f366ba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:30.311567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:30.311567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:30.311567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:30.311567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:30.311567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:30.311567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:30.311567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:30.311567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:30.311567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:30.311567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:30.311567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:30.311567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:16:30.311567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:16:30.311567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:16:30.311567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:16:30.313554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:30.313554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:30.313554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:30.313554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:30.313554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:30.313554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:30.313554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:30.313554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:30.313554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:30.313554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:30.313554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:30.313554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:30.313554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:30.313554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:30.313554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:30.313554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:30.313554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:16:30.314458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:16:30.314458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:30.314458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:16:30.314458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 617/1251] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:16:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:30.319549 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e477767-a972-4aa3-917c-e9d2d4e8304c None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:30.437790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e477767-a972-4aa3-917c-e9d2d4e8304c None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:30.484221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fb0fc7-f12c-4cbe-b6e1-e78b1374e50c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:30.485430 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fb0fc7-f12c-4cbe-b6e1-e78b1374e50c None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:30.485430 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fb0fc7-f12c-4cbe-b6e1-e78b1374e50c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:30.485732 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fb0fc7-f12c-4cbe-b6e1-e78b1374e50c None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:30.486412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fb0fc7-f12c-4cbe-b6e1-e78b1374e50c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:30.486714 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fb0fc7-f12c-4cbe-b6e1-e78b1374e50c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:30.488253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fb0fc7-f12c-4cbe-b6e1-e78b1374e50c None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:30.503340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 635/1252] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:16:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:16:30.509531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7375d2-03bc-4215-8200-aee948f366ba None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:30.627267 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7375d2-03bc-4215-8200-aee948f366ba None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:30.653983 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:30.664848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70aaf505-7294-42e4-b9cc-c0039947a568 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:30.665686 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70aaf505-7294-42e4-b9cc-c0039947a568 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:30.666173 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70aaf505-7294-42e4-b9cc-c0039947a568 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:30.666173 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70aaf505-7294-42e4-b9cc-c0039947a568 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:30.674989 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70aaf505-7294-42e4-b9cc-c0039947a568 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:30.676469 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70aaf505-7294-42e4-b9cc-c0039947a568 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:30.690523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-70aaf505-7294-42e4-b9cc-c0039947a568 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 30 04:16:30.690523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:30.690523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:30.690523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:30.690523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:30.690523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:30.690523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:30.690523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:30.690523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:30.690523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:30.690523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:30.690523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:30.690523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:16:30.690523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:16:30.690523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:16:30.690523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:16:30.691752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:30.691752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:30.691752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:16:30.691752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:16:30.691752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:16:30.691752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:16:30.691752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:30.691752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:30.691752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:30.691752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:30.691752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:30.691752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:30.691752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:30.691752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:30.691752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:30.691752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:30.691752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:30.691752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:30.692901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:30.692901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:30.692901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:16:30.692901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:16:30.692901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:16:30.692901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:16:30.692901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:16:30.692901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:16:30.692901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 30 04:16:30.692901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:16:30.692901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 618/1253] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:16:30 2022] GET /identity/v3/users/placement => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:30.697931 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fb0fc7-f12c-4cbe-b6e1-e78b1374e50c None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:30.767780 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:30.828026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fb0fc7-f12c-4cbe-b6e1-e78b1374e50c None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:30.871873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4239624f-d724-43c8-bcd2-0beab2f8809e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:30.872587 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4239624f-d724-43c8-bcd2-0beab2f8809e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:30.872936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4239624f-d724-43c8-bcd2-0beab2f8809e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:30.873322 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4239624f-d724-43c8-bcd2-0beab2f8809e None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:30.874065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4239624f-d724-43c8-bcd2-0beab2f8809e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:30.875336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4239624f-d724-43c8-bcd2-0beab2f8809e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:30.876587 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4239624f-d724-43c8-bcd2-0beab2f8809e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:30.897867 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 636/1254] 10.222.0.85 () {56 vars in 1143 bytes} [Tue Aug 30 04:16:30 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:30.904231 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70aaf505-7294-42e4-b9cc-c0039947a568 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:31.045937 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70aaf505-7294-42e4-b9cc-c0039947a568 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:31.092595 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb599cf4-be5f-4c9d-8a03-5c82bbb4056b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:31.093809 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb599cf4-be5f-4c9d-8a03-5c82bbb4056b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:31.094166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb599cf4-be5f-4c9d-8a03-5c82bbb4056b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:31.094455 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb599cf4-be5f-4c9d-8a03-5c82bbb4056b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:31.168727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:31.184160 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb599cf4-be5f-4c9d-8a03-5c82bbb4056b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:31.185691 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb599cf4-be5f-4c9d-8a03-5c82bbb4056b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:31.319974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 619/1255] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:16:30 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 04:16:31.326688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4239624f-d724-43c8-bcd2-0beab2f8809e None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:31.448866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4239624f-d724-43c8-bcd2-0beab2f8809e None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:31.471798 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:31.496229 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fff27ea-d19c-4a48-95b6-2719d8233fce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:31.497525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fff27ea-d19c-4a48-95b6-2719d8233fce None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:31.498235 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fff27ea-d19c-4a48-95b6-2719d8233fce None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:31.498793 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fff27ea-d19c-4a48-95b6-2719d8233fce None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:31.516107 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fff27ea-d19c-4a48-95b6-2719d8233fce None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:31.517887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fff27ea-d19c-4a48-95b6-2719d8233fce None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:31.541657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-2fff27ea-d19c-4a48-95b6-2719d8233fce None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:31.541657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:31.541657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:31.541657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:31.541657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:31.541657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:31.541657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:31.541657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:31.541657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:31.541657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:31.541657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:31.541657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:31.541657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:16:31.541657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:16:31.541657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:16:31.541657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:16:31.542860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:31.542860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:31.542860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:31.542860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:31.542860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:31.542860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:31.542860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:31.542860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:31.542860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:31.542860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:31.542860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:31.542860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:31.542860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:31.542860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:31.542860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:31.542860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:31.542860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:16:31.543883 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:16:31.543883 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:31.543883 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:31.545626 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 637/1256] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:16:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:31.551012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb599cf4-be5f-4c9d-8a03-5c82bbb4056b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:31.665882 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:31.684633 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb599cf4-be5f-4c9d-8a03-5c82bbb4056b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:31.749809 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5955f4a3-3d1b-4241-a23b-8b820829d2df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:31.750420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5955f4a3-3d1b-4241-a23b-8b820829d2df None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:31.750816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5955f4a3-3d1b-4241-a23b-8b820829d2df None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:31.751180 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5955f4a3-3d1b-4241-a23b-8b820829d2df None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:31.752082 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5955f4a3-3d1b-4241-a23b-8b820829d2df None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:31.752430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5955f4a3-3d1b-4241-a23b-8b820829d2df None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:31.754618 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5955f4a3-3d1b-4241-a23b-8b820829d2df None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:31.776503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 620/1257] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:16:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:31.782735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fff27ea-d19c-4a48-95b6-2719d8233fce None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:31.911161 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fff27ea-d19c-4a48-95b6-2719d8233fce None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:31.954285 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcf8639-493a-4edc-8ba7-5d0a33f7e36c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:31.954912 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcf8639-493a-4edc-8ba7-5d0a33f7e36c None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:31.955191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcf8639-493a-4edc-8ba7-5d0a33f7e36c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:31.955464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcf8639-493a-4edc-8ba7-5d0a33f7e36c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:31.962960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcf8639-493a-4edc-8ba7-5d0a33f7e36c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:31.964428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcf8639-493a-4edc-8ba7-5d0a33f7e36c None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:31.977113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-9bcf8639-493a-4edc-8ba7-5d0a33f7e36c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:16:31.977113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:31.977113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:31.977113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:31.977113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:31.977113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:31.977113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:31.977113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:31.977113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:31.977113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:31.977113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:31.977113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:31.977113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:16:31.977113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:16:31.977113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:16:31.978309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:16:31.978309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:31.978309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:31.978309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:31.978309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:31.978309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:31.978309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:31.978309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:31.978309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:31.978309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:31.978309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:31.978309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:31.978309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:31.978309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:31.978309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:31.978309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:31.978309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:31.979409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:16:31.979409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:16:31.979409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:16:31.979409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:16:31.979409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:16:31.979409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:16:31.979409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:16:31.979409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:31.979409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 638/1258] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:16:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:31.984366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5955f4a3-3d1b-4241-a23b-8b820829d2df None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:32.119556 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5955f4a3-3d1b-4241-a23b-8b820829d2df None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:32.160217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c99d032-4a7a-40c9-8fa8-17f832b89d84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:32.160828 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c99d032-4a7a-40c9-8fa8-17f832b89d84 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:32.161150 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c99d032-4a7a-40c9-8fa8-17f832b89d84 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:32.161400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c99d032-4a7a-40c9-8fa8-17f832b89d84 None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:32.162117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c99d032-4a7a-40c9-8fa8-17f832b89d84 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:32.162423 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c99d032-4a7a-40c9-8fa8-17f832b89d84 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:32.163875 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c99d032-4a7a-40c9-8fa8-17f832b89d84 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:32.176935 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 621/1259] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:16:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:32.183510 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcf8639-493a-4edc-8ba7-5d0a33f7e36c None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:32.219989 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:32.319575 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcf8639-493a-4edc-8ba7-5d0a33f7e36c None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:32.360358 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8191b052-4d56-44ba-b032-4e4638583d74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:32.361018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8191b052-4d56-44ba-b032-4e4638583d74 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:32.361305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8191b052-4d56-44ba-b032-4e4638583d74 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:32.361579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8191b052-4d56-44ba-b032-4e4638583d74 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:32.362320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8191b052-4d56-44ba-b032-4e4638583d74 None admin] RBAC: Adding query filter params (role.id=d17954e7ce544f0d90a512340232c60e, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=359d79cbfcad49beba2b495066824128) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:32.362628 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8191b052-4d56-44ba-b032-4e4638583d74 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:32.364439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8191b052-4d56-44ba-b032-4e4638583d74 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:32.389164 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 639/1260] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:16:32 2022] GET /identity/v3/role_assignments?user.id=359d79cbfcad49beba2b495066824128&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=d17954e7ce544f0d90a512340232c60e => generated 259 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:16:34.309828 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8c3c3e-4ae4-482b-8394-9ba0b29f080a None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:34.310442 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8c3c3e-4ae4-482b-8394-9ba0b29f080a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:34.310442 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8c3c3e-4ae4-482b-8394-9ba0b29f080a None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:34.312213 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 622/1261] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:16:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:16:34.317571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b157bbf9-7bed-4405-88eb-55fd636d95c9 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:34.318029 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b157bbf9-7bed-4405-88eb-55fd636d95c9 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:34.318140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b157bbf9-7bed-4405-88eb-55fd636d95c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:34.337266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-b157bbf9-7bed-4405-88eb-55fd636d95c9 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:16:34.338464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b157bbf9-7bed-4405-88eb-55fd636d95c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:16:34.405767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 640/1262] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:16:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:16:34.672149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8c3c3e-4ae4-482b-8394-9ba0b29f080a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:34.684404 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:34.788969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8c3c3e-4ae4-482b-8394-9ba0b29f080a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:34.823301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca859ed-5246-4ab9-bebb-8d725375ba48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:34.823927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca859ed-5246-4ab9-bebb-8d725375ba48 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:34.824202 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca859ed-5246-4ab9-bebb-8d725375ba48 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:34.824472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca859ed-5246-4ab9-bebb-8d725375ba48 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:34.834983 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca859ed-5246-4ab9-bebb-8d725375ba48 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:16:34.839869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-9ca859ed-5246-4ab9-bebb-8d725375ba48 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:16:34.839869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:34.839869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:34.839869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:34.839869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:34.839869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:34.839869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:34.839869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:34.839869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:34.839869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:34.839869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:34.839869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:34.839869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:16:34.839869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:16:34.839869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:16:34.839869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:16:34.840978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:16:34.840978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:16:34.840978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:16:34.840978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:16:34.840978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:34.840978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:34.840978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:34.840978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:34.840978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:34.840978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:34.840978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:34.840978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:34.840978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:34.840978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:34.840978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:34.840978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:34.842220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:34.842220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:34.842220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:34.842220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:34.842220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:16:34.842220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:16:34.842220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:16:34.842220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:16:34.842220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:16:34.842220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:16:34.842220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:16:34.842220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:16:34.842220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 623/1263] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:16:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 04:16:34.847598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b157bbf9-7bed-4405-88eb-55fd636d95c9 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:34.866280 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:34.945367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b157bbf9-7bed-4405-88eb-55fd636d95c9 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:34.979053 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9537cb-30ac-4f97-a218-3ab93a748d4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:34.979646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9537cb-30ac-4f97-a218-3ab93a748d4e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:34.979921 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9537cb-30ac-4f97-a218-3ab93a748d4e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:34.980188 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9537cb-30ac-4f97-a218-3ab93a748d4e None admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:34.980842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9537cb-30ac-4f97-a218-3ab93a748d4e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:34.981165 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9537cb-30ac-4f97-a218-3ab93a748d4e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:34.982510 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9537cb-30ac-4f97-a218-3ab93a748d4e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:34.990537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 641/1264] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:16:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:34.996273 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca859ed-5246-4ab9-bebb-8d725375ba48 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:35.062443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:35.095767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca859ed-5246-4ab9-bebb-8d725375ba48 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:35.130788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1907b0ae-9c6c-4485-93c5-db240ed561da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:35.131424 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1907b0ae-9c6c-4485-93c5-db240ed561da None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:35.131634 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1907b0ae-9c6c-4485-93c5-db240ed561da None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:35.131902 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1907b0ae-9c6c-4485-93c5-db240ed561da None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:35.143032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1907b0ae-9c6c-4485-93c5-db240ed561da None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:35.144492 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1907b0ae-9c6c-4485-93c5-db240ed561da None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:35.159317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-1907b0ae-9c6c-4485-93c5-db240ed561da None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:35.159317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:35.159317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:35.159317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:35.159317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:35.159317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:35.159317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:35.159317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:35.159317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:35.159317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:35.159317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:35.159317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:35.159317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:16:35.159317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:16:35.159317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:16:35.159317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:16:35.160476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:35.160476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:35.160476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:35.160476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:35.160476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:35.160476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:35.160476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:35.160476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:35.160476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:35.160476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:35.160476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:35.160476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:35.160476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:35.160476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:35.160476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:35.160476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:35.160476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:16:35.161419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:16:35.161419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:35.161419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:16:35.161419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 624/1265] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:16:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:35.166509 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9537cb-30ac-4f97-a218-3ab93a748d4e None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:35.257225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:35.277776 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9537cb-30ac-4f97-a218-3ab93a748d4e None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:35.311592 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8207aa04-c930-4d5b-b967-bf2c4c4300ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:35.312196 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8207aa04-c930-4d5b-b967-bf2c4c4300ca None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:35.312476 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8207aa04-c930-4d5b-b967-bf2c4c4300ca None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:35.312748 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8207aa04-c930-4d5b-b967-bf2c4c4300ca None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:35.313493 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8207aa04-c930-4d5b-b967-bf2c4c4300ca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:35.313796 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8207aa04-c930-4d5b-b967-bf2c4c4300ca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:35.315176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8207aa04-c930-4d5b-b967-bf2c4c4300ca None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:35.326932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 642/1266] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:16:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:35.333301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1907b0ae-9c6c-4485-93c5-db240ed561da None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:35.432127 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1907b0ae-9c6c-4485-93c5-db240ed561da None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:35.451391 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:35.466322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b253956-059b-41c9-9b8d-575bcb97a87c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:35.466958 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b253956-059b-41c9-9b8d-575bcb97a87c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:35.467239 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b253956-059b-41c9-9b8d-575bcb97a87c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:35.467515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b253956-059b-41c9-9b8d-575bcb97a87c None admin] PATH_INFO: `/v3/users/placement` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:35.476528 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b253956-059b-41c9-9b8d-575bcb97a87c None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:35.478024 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b253956-059b-41c9-9b8d-575bcb97a87c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:35.489241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-0b253956-059b-41c9-9b8d-575bcb97a87c None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 30 04:16:35.489241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:35.489241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:35.489241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:35.489241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:35.489241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:35.489241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:35.489241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:35.489241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:35.489241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:35.489241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:35.489241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:35.489241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:16:35.489241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:16:35.489241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:16:35.489241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:16:35.490371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:35.490371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:35.490371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:16:35.490371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:16:35.490371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:16:35.490371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:16:35.490371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:35.490371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:35.490371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:35.490371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:35.490371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:35.490371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:35.490371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:35.490371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:35.490371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:35.490371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:35.490371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:35.490371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:35.491522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:35.491522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:35.491522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:16:35.491522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:16:35.491522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:16:35.491522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:16:35.491522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:16:35.491522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:16:35.491522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 30 04:16:35.491522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:16:35.491522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 625/1267] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:16:35 2022] GET /identity/v3/users/placement => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:35.496347 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8207aa04-c930-4d5b-b967-bf2c4c4300ca None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:35.602575 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8207aa04-c930-4d5b-b967-bf2c4c4300ca None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:35.637875 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b081f956-6718-4d27-85d2-e0f671e3de60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:35.638613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b081f956-6718-4d27-85d2-e0f671e3de60 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:35.638945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b081f956-6718-4d27-85d2-e0f671e3de60 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:35.639216 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b081f956-6718-4d27-85d2-e0f671e3de60 None admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:35.640814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b081f956-6718-4d27-85d2-e0f671e3de60 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:35.641144 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b081f956-6718-4d27-85d2-e0f671e3de60 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:35.642648 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b081f956-6718-4d27-85d2-e0f671e3de60 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:35.652319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:35.662331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 643/1268] 10.222.0.85 () {56 vars in 1143 bytes} [Tue Aug 30 04:16:35 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:35.670860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b253956-059b-41c9-9b8d-575bcb97a87c None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:35.778038 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b253956-059b-41c9-9b8d-575bcb97a87c None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:35.812139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847b4873-fbae-49a4-8818-d4c7e4c04c24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:35.812783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847b4873-fbae-49a4-8818-d4c7e4c04c24 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:35.813086 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847b4873-fbae-49a4-8818-d4c7e4c04c24 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:35.813364 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847b4873-fbae-49a4-8818-d4c7e4c04c24 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:35.824820 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847b4873-fbae-49a4-8818-d4c7e4c04c24 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:35.826323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847b4873-fbae-49a4-8818-d4c7e4c04c24 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:35.841536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-847b4873-fbae-49a4-8818-d4c7e4c04c24 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:35.841536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:35.841536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:35.841536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:35.841536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:35.841536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:35.841536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:35.841536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:35.841536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:35.841536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:35.841536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:35.841536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:35.841536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:16:35.841536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:16:35.841536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:16:35.841536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:16:35.847770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:35.847770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:35.847770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:35.847770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:35.847770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:35.847770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:35.847770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:35.847770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:35.847770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:35.847770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:35.847770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:35.847770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:35.847770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:35.847770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:35.847770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:35.847770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:35.847770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:16:35.848917 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:16:35.848917 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:35.848917 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:16:35.848917 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 626/1269] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:16:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:35.849215 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b081f956-6718-4d27-85d2-e0f671e3de60 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:35.962379 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b081f956-6718-4d27-85d2-e0f671e3de60 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:35.997056 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7de626-3998-4075-a50e-070c7f6911fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:35.997809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7de626-3998-4075-a50e-070c7f6911fb None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:35.998179 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7de626-3998-4075-a50e-070c7f6911fb None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:35.998555 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7de626-3998-4075-a50e-070c7f6911fb None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:35.999967 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7de626-3998-4075-a50e-070c7f6911fb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:36.000270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7de626-3998-4075-a50e-070c7f6911fb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:36.001678 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7de626-3998-4075-a50e-070c7f6911fb None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:36.013478 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 644/1270] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:16:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:36.019184 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847b4873-fbae-49a4-8818-d4c7e4c04c24 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:36.051887 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:36.123691 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847b4873-fbae-49a4-8818-d4c7e4c04c24 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:36.157987 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac0bc19-f9b1-442e-ac48-64ab3d09f0bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:36.158648 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac0bc19-f9b1-442e-ac48-64ab3d09f0bc None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:36.158930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac0bc19-f9b1-442e-ac48-64ab3d09f0bc None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:36.159203 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac0bc19-f9b1-442e-ac48-64ab3d09f0bc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:36.165991 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac0bc19-f9b1-442e-ac48-64ab3d09f0bc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:36.167453 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac0bc19-f9b1-442e-ac48-64ab3d09f0bc None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:36.179459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-8ac0bc19-f9b1-442e-ac48-64ab3d09f0bc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:16:36.179459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:36.179459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:36.179459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:36.179459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:36.179459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:36.179459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:36.179459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:36.179459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:36.179459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:36.179459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:36.179459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:36.179459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:16:36.179459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:16:36.179459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:16:36.180521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:16:36.180521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:36.180521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:36.180521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:36.180521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:36.180521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:36.180521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:36.180521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:36.180521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:36.180521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:36.180521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:36.180521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:36.180521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:36.180521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:36.180521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:36.180521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:36.180521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:36.181810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:16:36.181810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:16:36.181810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:16:36.181810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:16:36.181810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:16:36.181810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:16:36.181810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:16:36.181810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:16:36.181810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 627/1271] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:16:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:36.186519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7de626-3998-4075-a50e-070c7f6911fb None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:36.250479 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:36.292201 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7de626-3998-4075-a50e-070c7f6911fb None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:36.325187 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d34c49-13a7-4ac2-b06b-7805bea85902 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:36.325783 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d34c49-13a7-4ac2-b06b-7805bea85902 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:36.326066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d34c49-13a7-4ac2-b06b-7805bea85902 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:36.326340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d34c49-13a7-4ac2-b06b-7805bea85902 None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:36.327025 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d34c49-13a7-4ac2-b06b-7805bea85902 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:36.328103 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d34c49-13a7-4ac2-b06b-7805bea85902 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:36.329371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d34c49-13a7-4ac2-b06b-7805bea85902 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:36.340925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 645/1272] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:16:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:36.346891 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac0bc19-f9b1-442e-ac48-64ab3d09f0bc None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:36.435814 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:36.445201 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac0bc19-f9b1-442e-ac48-64ab3d09f0bc None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:36.477992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4699e8e0-d857-4dc5-a571-f73d10d4f684 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:36.478635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4699e8e0-d857-4dc5-a571-f73d10d4f684 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:36.478914 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4699e8e0-d857-4dc5-a571-f73d10d4f684 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:36.479184 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4699e8e0-d857-4dc5-a571-f73d10d4f684 None admin] PATH_INFO: `/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/359d79cbfcad49beba2b495066824128/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:36.481161 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4699e8e0-d857-4dc5-a571-f73d10d4f684 None admin] RBAC: Authorizing `identity:create_grant(project_id=a860dcc3b2d64d1491c40710ec7423a5, user_id=359d79cbfcad49beba2b495066824128, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:36.482597 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4699e8e0-d857-4dc5-a571-f73d10d4f684 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:36.494838 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 628/1273] 10.222.0.85 () {58 vars in 1439 bytes} [Tue Aug 30 04:16:36 2022] PUT /identity/v3/projects/a860dcc3b2d64d1491c40710ec7423a5/users/359d79cbfcad49beba2b495066824128/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:16:38.371707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47774afc-03e9-4539-9150-69051a0d3048 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:38.372262 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47774afc-03e9-4539-9150-69051a0d3048 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:38.372262 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47774afc-03e9-4539-9150-69051a0d3048 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:38.373530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 646/1274] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:16:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:16:38.378845 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bac5cc5-aa7c-40e4-8943-0c9cd0cb239f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:38.379263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bac5cc5-aa7c-40e4-8943-0c9cd0cb239f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:38.379389 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bac5cc5-aa7c-40e4-8943-0c9cd0cb239f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:38.398129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-8bac5cc5-aa7c-40e4-8943-0c9cd0cb239f None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:16:38.399334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8bac5cc5-aa7c-40e4-8943-0c9cd0cb239f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:16:38.466752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 629/1275] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:16:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:16:38.733731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47774afc-03e9-4539-9150-69051a0d3048 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:38.847558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47774afc-03e9-4539-9150-69051a0d3048 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:38.865206 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:38.882791 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea72846f-43b1-415b-9d1a-d1da409e9020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:38.883427 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea72846f-43b1-415b-9d1a-d1da409e9020 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:38.883703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea72846f-43b1-415b-9d1a-d1da409e9020 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:38.883973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea72846f-43b1-415b-9d1a-d1da409e9020 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:38.895068 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea72846f-43b1-415b-9d1a-d1da409e9020 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 04:16:38.899879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-ea72846f-43b1-415b-9d1a-d1da409e9020 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:16:38.899879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:38.899879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:38.899879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:38.899879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:38.899879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:38.899879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:38.899879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:38.899879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:38.899879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:38.899879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:38.899879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:38.899879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 04:16:38.899879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 04:16:38.899879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 04:16:38.899879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 04:16:38.901101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 04:16:38.901101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 04:16:38.901101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 04:16:38.901101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 04:16:38.901101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:38.901101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:38.901101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:38.901101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:38.901101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:38.901101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:38.901101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:38.901101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:38.901101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:38.901101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:38.901101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:38.901101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:38.902227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:38.902227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:38.902227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:38.902227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:38.902227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 04:16:38.902227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 04:16:38.902227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 04:16:38.902227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 04:16:38.902227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 04:16:38.902227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 04:16:38.902227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 04:16:38.902227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:38.902227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 647/1276] 10.222.0.85 () {56 vars in 1096 bytes} [Tue Aug 30 04:16:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:38.907175 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bac5cc5-aa7c-40e4-8943-0c9cd0cb239f None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:38.989983 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:39.009514 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bac5cc5-aa7c-40e4-8943-0c9cd0cb239f None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:39.049355 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c8f2d9-25d9-42dc-bc0d-110ebc6f994b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:39.050168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c8f2d9-25d9-42dc-bc0d-110ebc6f994b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:39.050447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c8f2d9-25d9-42dc-bc0d-110ebc6f994b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:39.050713 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c8f2d9-25d9-42dc-bc0d-110ebc6f994b None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:39.051379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c8f2d9-25d9-42dc-bc0d-110ebc6f994b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:39.051701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c8f2d9-25d9-42dc-bc0d-110ebc6f994b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:39.053084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c8f2d9-25d9-42dc-bc0d-110ebc6f994b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:39.062457 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 630/1277] 10.222.0.85 () {56 vars in 1099 bytes} [Tue Aug 30 04:16:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:39.068318 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea72846f-43b1-415b-9d1a-d1da409e9020 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:39.168693 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea72846f-43b1-415b-9d1a-d1da409e9020 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:39.204068 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a66441-3936-44c9-8d2b-f6c6e64a3346 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:39.204702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a66441-3936-44c9-8d2b-f6c6e64a3346 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:39.204998 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a66441-3936-44c9-8d2b-f6c6e64a3346 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:39.205328 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a66441-3936-44c9-8d2b-f6c6e64a3346 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:39.215321 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a66441-3936-44c9-8d2b-f6c6e64a3346 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:39.216784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a66441-3936-44c9-8d2b-f6c6e64a3346 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:39.231392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-14a66441-3936-44c9-8d2b-f6c6e64a3346 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:39.231392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:39.231392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:39.231392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:39.231392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:39.231392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:39.231392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:39.231392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:39.231392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:39.231392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:39.231392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:39.231392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:39.231392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:16:39.231392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:16:39.231392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:16:39.231392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:16:39.233940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:39.233940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:39.233940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:39.233940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:39.233940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:39.233940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:39.233940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:39.233940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:39.233940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:39.233940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:39.233940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:39.233940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:39.233940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:39.233940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:39.233940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:39.233940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:39.233940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:16:39.235286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:16:39.235286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:39.235286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:39.235286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 648/1278] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:16:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:39.239474 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c8f2d9-25d9-42dc-bc0d-110ebc6f994b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:39.338660 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c8f2d9-25d9-42dc-bc0d-110ebc6f994b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:39.372834 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffaae6e4-7197-4d4b-adf6-2953700e19ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:39.373488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffaae6e4-7197-4d4b-adf6-2953700e19ee None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:39.373808 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffaae6e4-7197-4d4b-adf6-2953700e19ee None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:39.374084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffaae6e4-7197-4d4b-adf6-2953700e19ee None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:39.374732 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffaae6e4-7197-4d4b-adf6-2953700e19ee None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:39.375037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffaae6e4-7197-4d4b-adf6-2953700e19ee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:39.376407 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffaae6e4-7197-4d4b-adf6-2953700e19ee None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:39.388282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 631/1279] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:16:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:39.394040 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a66441-3936-44c9-8d2b-f6c6e64a3346 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:39.425184 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:39.495115 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a66441-3936-44c9-8d2b-f6c6e64a3346 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:39.528494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39add301-d343-4963-9168-25c3d9a4245c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:39.529175 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39add301-d343-4963-9168-25c3d9a4245c None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:39.529455 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39add301-d343-4963-9168-25c3d9a4245c None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:39.529724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39add301-d343-4963-9168-25c3d9a4245c None admin] PATH_INFO: `/v3/users/placement` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:39.537665 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39add301-d343-4963-9168-25c3d9a4245c None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:39.539117 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39add301-d343-4963-9168-25c3d9a4245c None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:39.551275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-39add301-d343-4963-9168-25c3d9a4245c None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 30 04:16:39.551275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:39.551275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:39.551275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:39.551275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:39.551275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:39.551275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:39.551275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:39.551275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:39.551275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:39.551275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:39.551275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:39.551275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 04:16:39.551275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 04:16:39.551275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 04:16:39.551275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 04:16:39.552316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:39.552316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:39.552316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 04:16:39.552316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:16:39.552316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 04:16:39.552316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 04:16:39.552316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:39.552316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:39.552316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:39.552316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:39.552316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:39.552316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:39.552316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:39.552316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:39.552316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:39.552316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:39.552316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:39.552316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:39.553320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:39.553320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:39.553320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 04:16:39.553320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 04:16:39.553320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 04:16:39.553320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 04:16:39.553320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 04:16:39.553320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 04:16:39.553320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 30 04:16:39.553320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:39.553320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 649/1280] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:16:39 2022] GET /identity/v3/users/placement => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:39.558222 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffaae6e4-7197-4d4b-adf6-2953700e19ee None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:39.562054 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:39.660755 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffaae6e4-7197-4d4b-adf6-2953700e19ee None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:39.700264 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846d7c46-501e-40f5-94ed-f65e56121ae3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:39.701336 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846d7c46-501e-40f5-94ed-f65e56121ae3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:39.701821 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846d7c46-501e-40f5-94ed-f65e56121ae3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:39.702665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846d7c46-501e-40f5-94ed-f65e56121ae3 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:39.703432 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846d7c46-501e-40f5-94ed-f65e56121ae3 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:39.703744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846d7c46-501e-40f5-94ed-f65e56121ae3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:39.705224 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846d7c46-501e-40f5-94ed-f65e56121ae3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:39.723399 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 632/1281] 10.222.0.85 () {56 vars in 1143 bytes} [Tue Aug 30 04:16:39 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:39.729317 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39add301-d343-4963-9168-25c3d9a4245c None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:39.830118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:39.840288 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39add301-d343-4963-9168-25c3d9a4245c None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:39.874553 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9123e0e2-4e8f-4015-9ae7-ad67874e4f65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:39.875153 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9123e0e2-4e8f-4015-9ae7-ad67874e4f65 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:39.875433 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9123e0e2-4e8f-4015-9ae7-ad67874e4f65 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:39.875707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9123e0e2-4e8f-4015-9ae7-ad67874e4f65 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:39.947381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9123e0e2-4e8f-4015-9ae7-ad67874e4f65 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:39.948834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9123e0e2-4e8f-4015-9ae7-ad67874e4f65 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:40.041855 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:40.052999 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 650/1282] 10.222.0.85 () {58 vars in 1282 bytes} [Tue Aug 30 04:16:39 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 04:16:40.058836 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846d7c46-501e-40f5-94ed-f65e56121ae3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:40.123329 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:40.162045 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846d7c46-501e-40f5-94ed-f65e56121ae3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:40.195679 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c97e65-c191-4f49-a4e6-f265d23b5000 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:40.196313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c97e65-c191-4f49-a4e6-f265d23b5000 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:40.196620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c97e65-c191-4f49-a4e6-f265d23b5000 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:40.196862 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c97e65-c191-4f49-a4e6-f265d23b5000 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:40.206767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c97e65-c191-4f49-a4e6-f265d23b5000 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:40.208147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c97e65-c191-4f49-a4e6-f265d23b5000 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:40.222920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-29c97e65-c191-4f49-a4e6-f265d23b5000 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:40.222920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:40.222920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:40.222920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:40.222920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:40.222920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:40.222920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:40.222920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:40.222920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:40.222920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:40.222920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:40.222920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:40.222920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 04:16:40.222920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 04:16:40.222920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 04:16:40.222920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 04:16:40.228739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:40.228739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:40.228739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:40.228739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:40.228739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:40.228739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:40.228739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:40.228739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:40.228739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:40.228739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:40.228739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:40.228739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:40.228739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:40.228739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:40.228739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:40.228739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:40.228739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 04:16:40.229910 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 04:16:40.229910 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 04:16:40.229910 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:16:40.229910 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 633/1283] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:16:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:40.230208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9123e0e2-4e8f-4015-9ae7-ad67874e4f65 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:40.327255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9123e0e2-4e8f-4015-9ae7-ad67874e4f65 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:40.361050 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2c3c04-b3ef-48c8-b0d2-07ce0f18ad4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:40.361667 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2c3c04-b3ef-48c8-b0d2-07ce0f18ad4f None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:40.361944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2c3c04-b3ef-48c8-b0d2-07ce0f18ad4f None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:40.362212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2c3c04-b3ef-48c8-b0d2-07ce0f18ad4f None admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:40.362846 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2c3c04-b3ef-48c8-b0d2-07ce0f18ad4f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:40.363143 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2c3c04-b3ef-48c8-b0d2-07ce0f18ad4f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:40.364500 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2c3c04-b3ef-48c8-b0d2-07ce0f18ad4f None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:40.375480 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 651/1284] 10.222.0.85 () {56 vars in 1109 bytes} [Tue Aug 30 04:16:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:16:40.381463 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c97e65-c191-4f49-a4e6-f265d23b5000 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:40.481850 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c97e65-c191-4f49-a4e6-f265d23b5000 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:40.495260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:40.515706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025062f4-7d5b-45eb-84a3-e7de451731d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:40.516333 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025062f4-7d5b-45eb-84a3-e7de451731d7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:40.516612 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025062f4-7d5b-45eb-84a3-e7de451731d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:40.516882 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025062f4-7d5b-45eb-84a3-e7de451731d7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:40.523402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025062f4-7d5b-45eb-84a3-e7de451731d7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:40.524837 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025062f4-7d5b-45eb-84a3-e7de451731d7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:40.537627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-025062f4-7d5b-45eb-84a3-e7de451731d7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:16:40.537627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:40.537627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:40.537627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:40.537627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:40.537627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:40.537627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:40.537627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:40.537627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:40.537627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:40.537627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:40.537627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:40.537627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 04:16:40.537627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 04:16:40.537627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 04:16:40.538759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 04:16:40.538759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:40.538759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:40.538759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:40.538759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:40.538759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:40.538759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:40.538759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:40.538759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:40.538759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:40.538759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:40.538759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:40.538759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:40.538759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:40.538759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:40.538759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:40.538759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:40.539872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 04:16:40.539872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 04:16:40.539872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 04:16:40.539872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 04:16:40.539872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 04:16:40.539872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 04:16:40.539872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 04:16:40.539872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Aug 30 04:16:40.539872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 634/1285] 10.222.0.85 () {56 vars in 1111 bytes} [Tue Aug 30 04:16:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:40.547716 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2c3c04-b3ef-48c8-b0d2-07ce0f18ad4f None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:40.598265 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:40.648314 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2c3c04-b3ef-48c8-b0d2-07ce0f18ad4f None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:40.687038 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03398445-8b98-4b4d-b3fc-6684a7a02751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:40.687751 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03398445-8b98-4b4d-b3fc-6684a7a02751 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:40.688026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03398445-8b98-4b4d-b3fc-6684a7a02751 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:40.688296 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03398445-8b98-4b4d-b3fc-6684a7a02751 None admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:40.689008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03398445-8b98-4b4d-b3fc-6684a7a02751 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:40.689374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03398445-8b98-4b4d-b3fc-6684a7a02751 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:40.690794 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03398445-8b98-4b4d-b3fc-6684a7a02751 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:40.703165 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 652/1286] 10.222.0.85 () {56 vars in 1148 bytes} [Tue Aug 30 04:16:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:40.709252 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025062f4-7d5b-45eb-84a3-e7de451731d7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:40.807190 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025062f4-7d5b-45eb-84a3-e7de451731d7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:40.840380 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc6b318-2080-4e6e-8713-9f6ab2b482e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:40.840976 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc6b318-2080-4e6e-8713-9f6ab2b482e3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:40.841278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc6b318-2080-4e6e-8713-9f6ab2b482e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:40.841546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc6b318-2080-4e6e-8713-9f6ab2b482e3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:40.842266 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc6b318-2080-4e6e-8713-9f6ab2b482e3 None admin] RBAC: Adding query filter params (role.id=d17954e7ce544f0d90a512340232c60e, scope.project.id=a860dcc3b2d64d1491c40710ec7423a5, user.id=359d79cbfcad49beba2b495066824128) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:40.843138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc6b318-2080-4e6e-8713-9f6ab2b482e3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:40.844643 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc6b318-2080-4e6e-8713-9f6ab2b482e3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:40.851226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 635/1287] 10.222.0.85 () {56 vars in 1374 bytes} [Tue Aug 30 04:16:40 2022] GET /identity/v3/role_assignments?user.id=359d79cbfcad49beba2b495066824128&scope.project.id=a860dcc3b2d64d1491c40710ec7423a5&role.id=d17954e7ce544f0d90a512340232c60e => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:42.794843 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152f565b-3499-4f91-8e12-fd9bddac06f9 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:42.795381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152f565b-3499-4f91-8e12-fd9bddac06f9 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:42.795381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152f565b-3499-4f91-8e12-fd9bddac06f9 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:42.797309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 653/1288] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:16:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:16:42.804226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbaacfa-b8ed-4d24-9b9c-d71518bd31c7 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:42.804712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbaacfa-b8ed-4d24-9b9c-d71518bd31c7 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:42.805242 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbaacfa-b8ed-4d24-9b9c-d71518bd31c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:42.825946 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-0fbaacfa-b8ed-4d24-9b9c-d71518bd31c7 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:16:42.827240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0fbaacfa-b8ed-4d24-9b9c-d71518bd31c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:16:42.857731 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:42.900310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 636/1289] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:16:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 04:16:43.179310 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152f565b-3499-4f91-8e12-fd9bddac06f9 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:43.287232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152f565b-3499-4f91-8e12-fd9bddac06f9 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:43.299307 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:43.324259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1776750-cd0b-4c77-974d-a8c3a9a5c268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:43.324892 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1776750-cd0b-4c77-974d-a8c3a9a5c268 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:43.325208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1776750-cd0b-4c77-974d-a8c3a9a5c268 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:43.325505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1776750-cd0b-4c77-974d-a8c3a9a5c268 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:43.326095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1776750-cd0b-4c77-974d-a8c3a9a5c268 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:43.327464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1776750-cd0b-4c77-974d-a8c3a9a5c268 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:43.336643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-d1776750-cd0b-4c77-974d-a8c3a9a5c268 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 04:16:43.336643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:43.336643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:16:43.336643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:16:43.336643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:16:43.336643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:16:43.336643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:16:43.336643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:16:43.336643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 04:16:43.336643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:43.336643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:16:43.336643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:43.336643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:43.336643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:43.336643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:43.336643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:43.336643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:43.337944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:43.337944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:43.337944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:43.337944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:43.337944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:43.337944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:43.337944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:16:43.337944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:16:43.337944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:16:43.337944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:16:43.337944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:43.337944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:43.337944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:43.337944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:43.337944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:43.337944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:43.338972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:43.338972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:43.338972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:43.338972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:43.338972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:43.338972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:43.338972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:43.338972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:43.338972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:43.338972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:43.338972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:16:43.338972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:16:43.338972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 04:16:43.338972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:43.338972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 654/1290] 10.222.0.85 () {56 vars in 1117 bytes} [Tue Aug 30 04:16:43 2022] GET /identity/v3/services/placement => generated 90 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:43.344785 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbaacfa-b8ed-4d24-9b9c-d71518bd31c7 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:43.447466 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbaacfa-b8ed-4d24-9b9c-d71518bd31c7 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:43.487330 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf2c65d-5163-45e4-b408-2726c7a02708 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:43.487934 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf2c65d-5163-45e4-b408-2726c7a02708 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:43.488209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf2c65d-5163-45e4-b408-2726c7a02708 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:43.488478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf2c65d-5163-45e4-b408-2726c7a02708 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:43.489137 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf2c65d-5163-45e4-b408-2726c7a02708 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:43.489437 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf2c65d-5163-45e4-b408-2726c7a02708 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:43.490799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf2c65d-5163-45e4-b408-2726c7a02708 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:43.495602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 637/1291] 10.222.0.85 () {56 vars in 1116 bytes} [Tue Aug 30 04:16:43 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:16:43.501455 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1776750-cd0b-4c77-974d-a8c3a9a5c268 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:43.618090 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1776750-cd0b-4c77-974d-a8c3a9a5c268 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:43.655839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef0301e-31e6-4862-8546-af5e14d75a1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:43.656451 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef0301e-31e6-4862-8546-af5e14d75a1e None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:43.656733 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef0301e-31e6-4862-8546-af5e14d75a1e None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:43.657005 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef0301e-31e6-4862-8546-af5e14d75a1e None admin] PATH_INFO: `/v3/services` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:43.657696 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef0301e-31e6-4862-8546-af5e14d75a1e None admin] RBAC: Adding query filter params (type=placement) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:43.658000 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef0301e-31e6-4862-8546-af5e14d75a1e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:43.659409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef0301e-31e6-4862-8546-af5e14d75a1e None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:43.666097 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 655/1292] 10.222.0.85 () {56 vars in 1116 bytes} [Tue Aug 30 04:16:43 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:45.588971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92c1096-4711-49fd-8adb-d96ebd8ad982 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:45.589574 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92c1096-4711-49fd-8adb-d96ebd8ad982 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:45.589574 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92c1096-4711-49fd-8adb-d96ebd8ad982 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:45.590893 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 638/1293] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:16:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:16:45.596066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7593803-cd70-410b-8f01-9282c9259a4f None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:45.596477 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7593803-cd70-410b-8f01-9282c9259a4f None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:45.596610 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7593803-cd70-410b-8f01-9282c9259a4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:45.614337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-f7593803-cd70-410b-8f01-9282c9259a4f None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:16:45.615506 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f7593803-cd70-410b-8f01-9282c9259a4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:16:45.646268 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:45.688375 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 656/1294] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:16:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:16:45.951826 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92c1096-4711-49fd-8adb-d96ebd8ad982 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:45.982012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:46.053821 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92c1096-4711-49fd-8adb-d96ebd8ad982 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:46.086503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06920762-354c-42a8-b1f1-874a049faf0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:46.087105 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06920762-354c-42a8-b1f1-874a049faf0b None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:46.087383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06920762-354c-42a8-b1f1-874a049faf0b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:46.087652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06920762-354c-42a8-b1f1-874a049faf0b None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:46.088308 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06920762-354c-42a8-b1f1-874a049faf0b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:46.089692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06920762-354c-42a8-b1f1-874a049faf0b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:46.103978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 639/1295] 10.222.0.85 () {60 vars in 1141 bytes} [Tue Aug 30 04:16:45 2022] POST /identity/v3/services => generated 246 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:16:48.022874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b0a4ce-6474-491a-b5fa-ab40f1be306d None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:48.023447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b0a4ce-6474-491a-b5fa-ab40f1be306d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:48.023447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b0a4ce-6474-491a-b5fa-ab40f1be306d None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:48.024772 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 657/1296] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:16:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:16:48.029817 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a5985e-f2ae-4dbc-85d5-63f234684250 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:48.030260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a5985e-f2ae-4dbc-85d5-63f234684250 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:48.030396 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a5985e-f2ae-4dbc-85d5-63f234684250 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:48.048583 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-47a5985e-f2ae-4dbc-85d5-63f234684250 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:16:48.049798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-47a5985e-f2ae-4dbc-85d5-63f234684250 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:16:48.120384 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 640/1297] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:16:48 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:16:48.390596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b0a4ce-6474-491a-b5fa-ab40f1be306d None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:48.490895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b0a4ce-6474-491a-b5fa-ab40f1be306d None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:48.525208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f2fd83-a8d4-4a4e-9e18-0fdce52a4f7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:48.525830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f2fd83-a8d4-4a4e-9e18-0fdce52a4f7a None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:48.526104 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f2fd83-a8d4-4a4e-9e18-0fdce52a4f7a None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:48.526372 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f2fd83-a8d4-4a4e-9e18-0fdce52a4f7a None admin] PATH_INFO: `/v3/services/placement` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:48.526953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f2fd83-a8d4-4a4e-9e18-0fdce52a4f7a None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:48.528312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f2fd83-a8d4-4a4e-9e18-0fdce52a4f7a None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:48.537119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-96f2fd83-a8d4-4a4e-9e18-0fdce52a4f7a None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 04:16:48.537119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:48.537119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:16:48.537119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:16:48.537119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:16:48.537119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:16:48.537119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:16:48.537119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:16:48.537119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 04:16:48.537119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:48.537119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:16:48.537119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:48.537119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:48.537119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:48.537119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:48.537119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:48.537119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:48.541275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:48.541275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:48.541275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:48.541275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:48.541275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:48.541275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:48.541275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:16:48.541275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:16:48.541275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:16:48.541275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:16:48.541275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:48.541275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:48.541275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:48.541275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:48.541275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:48.541275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:48.542292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:48.542292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:48.542292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:48.542292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:48.542292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:48.542292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:48.542292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:48.542292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:48.542292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:48.542292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:48.542292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:16:48.542292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:16:48.542292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 04:16:48.542292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:48.542292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 658/1298] 10.222.0.85 () {56 vars in 1117 bytes} [Tue Aug 30 04:16:48 2022] GET /identity/v3/services/placement => generated 90 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:48.544329 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a5985e-f2ae-4dbc-85d5-63f234684250 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:48.563207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:48.643725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a5985e-f2ae-4dbc-85d5-63f234684250 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:48.683575 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443d69e7-64a5-4130-a4c8-acafe3f677f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:48.684506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443d69e7-64a5-4130-a4c8-acafe3f677f4 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:48.685069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443d69e7-64a5-4130-a4c8-acafe3f677f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:48.685662 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443d69e7-64a5-4130-a4c8-acafe3f677f4 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:48.687310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443d69e7-64a5-4130-a4c8-acafe3f677f4 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:48.687872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443d69e7-64a5-4130-a4c8-acafe3f677f4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:48.690220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443d69e7-64a5-4130-a4c8-acafe3f677f4 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:48.698107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 641/1299] 10.222.0.85 () {56 vars in 1116 bytes} [Tue Aug 30 04:16:48 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:48.704250 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f2fd83-a8d4-4a4e-9e18-0fdce52a4f7a None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:48.742983 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:48.812506 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f2fd83-a8d4-4a4e-9e18-0fdce52a4f7a None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:48.844703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c900c0-b818-492b-9d08-010b717ec965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:48.845330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c900c0-b818-492b-9d08-010b717ec965 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:48.845611 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c900c0-b818-492b-9d08-010b717ec965 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:48.845882 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c900c0-b818-492b-9d08-010b717ec965 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:48.846582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c900c0-b818-492b-9d08-010b717ec965 None admin] RBAC: Adding query filter params (interface=public, service_id=19a644baada74f8185e9be7d66b1c7a0, region_id=RegionOne) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:48.846890 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c900c0-b818-492b-9d08-010b717ec965 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:48.848245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c900c0-b818-492b-9d08-010b717ec965 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:48.853253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 659/1300] 10.222.0.85 () {56 vars in 1251 bytes} [Tue Aug 30 04:16:48 2022] GET /identity/v3/endpoints?service_id=19a644baada74f8185e9be7d66b1c7a0&interface=public®ion_id=RegionOne => generated 194 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:48.859420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443d69e7-64a5-4130-a4c8-acafe3f677f4 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:48.940719 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:48.959361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443d69e7-64a5-4130-a4c8-acafe3f677f4 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:48.993044 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9311743b-5140-4c38-8956-18fafb6d20d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:48.993674 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9311743b-5140-4c38-8956-18fafb6d20d1 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:48.993954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9311743b-5140-4c38-8956-18fafb6d20d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:48.994227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9311743b-5140-4c38-8956-18fafb6d20d1 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:48.994831 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9311743b-5140-4c38-8956-18fafb6d20d1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:48.996196 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9311743b-5140-4c38-8956-18fafb6d20d1 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:49.003768 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 642/1301] 10.222.0.85 () {56 vars in 1087 bytes} [Tue Aug 30 04:16:48 2022] GET /identity/v3/services => generated 2191 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:16:50.878675 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff3a1c7-3cbf-4cfb-8f44-9a0f67f5e3fb None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:50.879242 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff3a1c7-3cbf-4cfb-8f44-9a0f67f5e3fb None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:50.879242 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff3a1c7-3cbf-4cfb-8f44-9a0f67f5e3fb None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:50.880597 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 660/1302] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:16:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:16:50.885807 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66508946-6f24-47d9-8d76-a5d8c1bf1815 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:50.886257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66508946-6f24-47d9-8d76-a5d8c1bf1815 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:50.886352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66508946-6f24-47d9-8d76-a5d8c1bf1815 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:50.905691 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-66508946-6f24-47d9-8d76-a5d8c1bf1815 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:16:50.906884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-66508946-6f24-47d9-8d76-a5d8c1bf1815 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:16:50.973014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 643/1303] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:16:50 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 04:16:51.239821 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff3a1c7-3cbf-4cfb-8f44-9a0f67f5e3fb None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:51.332891 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:51.340133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff3a1c7-3cbf-4cfb-8f44-9a0f67f5e3fb None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:51.374526 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a58c0a-92d4-42b6-aeca-2e9b7fb1b917 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:51.375155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a58c0a-92d4-42b6-aeca-2e9b7fb1b917 None admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:51.375434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a58c0a-92d4-42b6-aeca-2e9b7fb1b917 None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:51.375706 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a58c0a-92d4-42b6-aeca-2e9b7fb1b917 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:51.376292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a58c0a-92d4-42b6-aeca-2e9b7fb1b917 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:51.377695 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a58c0a-92d4-42b6-aeca-2e9b7fb1b917 None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:51.386755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application [None req-b5a58c0a-92d4-42b6-aeca-2e9b7fb1b917 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 04:16:51.386755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:51.386755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 04:16:51.386755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 04:16:51.386755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 04:16:51.386755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 04:16:51.386755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 04:16:51.386755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:16:51.386755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 04:16:51.386755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:51.386755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 04:16:51.386755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:51.386755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 04:16:51.386755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 04:16:51.386755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 04:16:51.386755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 04:16:51.386755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 04:16:51.387886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 04:16:51.387886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 04:16:51.387886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 04:16:51.387886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 04:16:51.387886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 04:16:51.387886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 04:16:51.387886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 04:16:51.387886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 04:16:51.387886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 04:16:51.387886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 04:16:51.387886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 04:16:51.387886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 04:16:51.387886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 04:16:51.387886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 04:16:51.387886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 04:16:51.387886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 04:16:51.388872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 04:16:51.388872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application with Lock( Aug 30 04:16:51.388872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 04:16:51.388872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self._enter() Aug 30 04:16:51.388872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 04:16:51.388872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 04:16:51.388872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 04:16:51.388872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application return self.creator() Aug 30 04:16:51.388872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 04:16:51.388872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application created_value = creator( Aug 30 04:16:51.388872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 04:16:51.388872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 04:16:51.388872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 04:16:51.388872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: ERROR keystone.server.flask.application Aug 30 04:16:51.388872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 661/1304] 10.222.0.85 () {56 vars in 1117 bytes} [Tue Aug 30 04:16:51 2022] GET /identity/v3/services/placement => generated 90 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 04:16:51.393910 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66508946-6f24-47d9-8d76-a5d8c1bf1815 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:51.467801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:16:51.490601 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66508946-6f24-47d9-8d76-a5d8c1bf1815 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:51.524094 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81491275-166d-47b3-ac6f-a75555ab369c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:51.524703 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81491275-166d-47b3-ac6f-a75555ab369c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:51.524981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81491275-166d-47b3-ac6f-a75555ab369c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:51.525284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81491275-166d-47b3-ac6f-a75555ab369c None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:51.525916 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81491275-166d-47b3-ac6f-a75555ab369c None admin] RBAC: Adding query filter params (name=placement) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:16:51.526218 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81491275-166d-47b3-ac6f-a75555ab369c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:51.527593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81491275-166d-47b3-ac6f-a75555ab369c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:51.533367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 644/1305] 10.222.0.85 () {56 vars in 1116 bytes} [Tue Aug 30 04:16:51 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:16:51.539188 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a58c0a-92d4-42b6-aeca-2e9b7fb1b917 None admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:16:51.635631 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a58c0a-92d4-42b6-aeca-2e9b7fb1b917 None admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:16:51.669658 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c652d55-edab-4369-8d06-af2e6675f6ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:16:51.670291 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c652d55-edab-4369-8d06-af2e6675f6ac None admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:16:51.670624 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c652d55-edab-4369-8d06-af2e6675f6ac None admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:16:51.670896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c652d55-edab-4369-8d06-af2e6675f6ac None admin] PATH_INFO: `/v3/endpoints` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:16:51.671567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c652d55-edab-4369-8d06-af2e6675f6ac None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:16:51.672967 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c652d55-edab-4369-8d06-af2e6675f6ac None admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:16:51.694776 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 662/1306] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:16:51 2022] POST /identity/v3/endpoints => generated 331 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:17:01.529872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=30 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:17:01.689654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:17:56.236759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae153df7-d435-4c01-9c4a-e4ba58f144a9 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:17:56.237967 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae153df7-d435-4c01-9c4a-e4ba58f144a9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:17:56.238535 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae153df7-d435-4c01-9c4a-e4ba58f144a9 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:17:56.247498 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a37ef96-c4d6-4358-9620-34280d4235b3 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:17:56.247920 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a37ef96-c4d6-4358-9620-34280d4235b3 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:17:56.248170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a37ef96-c4d6-4358-9620-34280d4235b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:17:56.250775 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 645/1307] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:17:56 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:17:56.390256 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-2a37ef96-c4d6-4358-9620-34280d4235b3 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:17:56.392414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-2a37ef96-c4d6-4358-9620-34280d4235b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:17:56.557942 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 663/1308] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:17:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:17:56.861325 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1561c1e8-8d0c-4125-ae9f-b2cddd497b97 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:17:56.861656 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1561c1e8-8d0c-4125-ae9f-b2cddd497b97 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:17:56.861885 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1561c1e8-8d0c-4125-ae9f-b2cddd497b97 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:17:56.863325 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 646/1309] 10.222.0.85 () {54 vars in 965 bytes} [Tue Aug 30 04:17:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:17:56.870046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deda9749-46eb-4231-87e6-bd73a952bf7c None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:17:56.870391 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deda9749-46eb-4231-87e6-bd73a952bf7c None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:17:56.870608 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deda9749-46eb-4231-87e6-bd73a952bf7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:17:56.953735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-deda9749-46eb-4231-87e6-bd73a952bf7c None None] MFA Rules not processed for user `5c2edd77896a4534888d6758ec0ee0bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:17:56.954944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-deda9749-46eb-4231-87e6-bd73a952bf7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:17:56.977010 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:17:57.038992 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 664/1310] 10.222.0.85 () {58 vars in 1062 bytes} [Tue Aug 30 04:17:56 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:17:57.049555 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1561c1e8-8d0c-4125-ae9f-b2cddd497b97 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:17:57.051320 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1561c1e8-8d0c-4125-ae9f-b2cddd497b97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:17:57.170299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1561c1e8-8d0c-4125-ae9f-b2cddd497b97 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:17:57.248418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d466b7-5e99-497d-9818-0ba02f87712b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:17:57.249112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d466b7-5e99-497d-9818-0ba02f87712b service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:17:57.249418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d466b7-5e99-497d-9818-0ba02f87712b service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:17:57.249609 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d466b7-5e99-497d-9818-0ba02f87712b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:17:57.251183 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d4d466b7-5e99-497d-9818-0ba02f87712b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:17:57.311184 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:17:57.368987 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d466b7-5e99-497d-9818-0ba02f87712b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:17:57.370390 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d466b7-5e99-497d-9818-0ba02f87712b service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:17:57.390207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 647/1311] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:17:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:17:59.647583 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4f871a-a5d1-481d-9bde-62f1a66a3029 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:17:59.648094 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4f871a-a5d1-481d-9bde-62f1a66a3029 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:17:59.648208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4f871a-a5d1-481d-9bde-62f1a66a3029 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:17:59.649488 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 665/1312] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:17:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:17:59.655046 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bcc330-acfc-4677-bfbd-d1c87228e5f9 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:17:59.655471 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bcc330-acfc-4677-bfbd-d1c87228e5f9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:17:59.655593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bcc330-acfc-4677-bfbd-d1c87228e5f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:17:59.734132 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-30bcc330-acfc-4677-bfbd-d1c87228e5f9 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:17:59.735324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-30bcc330-acfc-4677-bfbd-d1c87228e5f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:17:59.829237 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 648/1313] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:17:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:18:00.108809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4f871a-a5d1-481d-9bde-62f1a66a3029 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:18:00.111084 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-fb4f871a-a5d1-481d-9bde-62f1a66a3029 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:00.215606 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4f871a-a5d1-481d-9bde-62f1a66a3029 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:18:00.288916 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:00.291409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be68492-c2a9-4773-a91d-493bd370619e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:18:00.292042 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be68492-c2a9-4773-a91d-493bd370619e service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:00.292504 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be68492-c2a9-4773-a91d-493bd370619e service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:00.293553 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be68492-c2a9-4773-a91d-493bd370619e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:00.295642 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-3be68492-c2a9-4773-a91d-493bd370619e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:00.412133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be68492-c2a9-4773-a91d-493bd370619e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:18:00.413562 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be68492-c2a9-4773-a91d-493bd370619e service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:18:00.432335 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 666/1314] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:18:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:18:02.653451 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7051ab-ab51-4a6f-95a7-f18b75122474 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:02.654007 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7051ab-ab51-4a6f-95a7-f18b75122474 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:02.654007 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7051ab-ab51-4a6f-95a7-f18b75122474 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:02.655342 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 649/1315] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:18:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:18:02.660502 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3909c4fe-05c2-4333-89f8-a83ade0a2ef3 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:02.660881 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3909c4fe-05c2-4333-89f8-a83ade0a2ef3 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:02.661093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3909c4fe-05c2-4333-89f8-a83ade0a2ef3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:02.728450 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-3909c4fe-05c2-4333-89f8-a83ade0a2ef3 None None] MFA Rules not processed for user `6a1eb60c46aa4d27b91657cd7f9fa60f`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:02.730074 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-3909c4fe-05c2-4333-89f8-a83ade0a2ef3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:02.781395 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:02.825532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 667/1316] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:18:02 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:18:03.115349 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7051ab-ab51-4a6f-95a7-f18b75122474 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:18:03.116932 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2d7051ab-ab51-4a6f-95a7-f18b75122474 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:03.127027 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:03.222315 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7051ab-ab51-4a6f-95a7-f18b75122474 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:18:03.298172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfadd58-0b20-42a3-b0e2-36227cdceaa3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:18:03.298777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfadd58-0b20-42a3-b0e2-36227cdceaa3 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:03.299057 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfadd58-0b20-42a3-b0e2-36227cdceaa3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:03.299329 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfadd58-0b20-42a3-b0e2-36227cdceaa3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:03.300969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bbfadd58-0b20-42a3-b0e2-36227cdceaa3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:03.363042 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:03.409862 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbfadd58-0b20-42a3-b0e2-36227cdceaa3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:18:03.411321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbfadd58-0b20-42a3-b0e2-36227cdceaa3 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:18:03.431101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 650/1317] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:18:03 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:18:05.889846 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db0ca9e-d95c-4df0-98a2-cfd86f41411d None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:05.890361 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db0ca9e-d95c-4df0-98a2-cfd86f41411d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:05.890361 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db0ca9e-d95c-4df0-98a2-cfd86f41411d None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:05.891633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 668/1318] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:18:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:18:05.896701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfbe792-c661-4db6-a7ef-87f4f7c22e8f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:05.897125 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfbe792-c661-4db6-a7ef-87f4f7c22e8f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:05.897274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfbe792-c661-4db6-a7ef-87f4f7c22e8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:05.961592 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-6cfbe792-c661-4db6-a7ef-87f4f7c22e8f None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:05.962815 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6cfbe792-c661-4db6-a7ef-87f4f7c22e8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:06.057318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 651/1319] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:18:05 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:18:06.341766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db0ca9e-d95c-4df0-98a2-cfd86f41411d None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:18:06.343320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-3db0ca9e-d95c-4df0-98a2-cfd86f41411d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:06.445717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db0ca9e-d95c-4df0-98a2-cfd86f41411d None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:18:06.518552 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631049eb-d36a-443c-9a01-c0f6414f4985 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:18:06.519156 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631049eb-d36a-443c-9a01-c0f6414f4985 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:06.519432 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631049eb-d36a-443c-9a01-c0f6414f4985 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:06.519701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631049eb-d36a-443c-9a01-c0f6414f4985 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:06.521322 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-631049eb-d36a-443c-9a01-c0f6414f4985 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:06.529576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:06.629159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631049eb-d36a-443c-9a01-c0f6414f4985 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:18:06.630519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631049eb-d36a-443c-9a01-c0f6414f4985 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:18:06.648757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 669/1320] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:18:06 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:18:09.228227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00062fc4-d41f-4cf9-b377-b1325261eb7b None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:09.228903 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00062fc4-d41f-4cf9-b377-b1325261eb7b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:09.228903 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00062fc4-d41f-4cf9-b377-b1325261eb7b None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:09.230355 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 652/1321] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:18:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:18:09.235287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfd8c31-8d34-4a7e-b3a5-964a31f0a7ac None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:09.235754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfd8c31-8d34-4a7e-b3a5-964a31f0a7ac None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:09.235864 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfd8c31-8d34-4a7e-b3a5-964a31f0a7ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:09.301720 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-adfd8c31-8d34-4a7e-b3a5-964a31f0a7ac None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:09.302931 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-adfd8c31-8d34-4a7e-b3a5-964a31f0a7ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:09.360430 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:09.400711 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 670/1322] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:18:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:18:09.688936 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00062fc4-d41f-4cf9-b377-b1325261eb7b None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:18:09.690541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-00062fc4-d41f-4cf9-b377-b1325261eb7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:09.710876 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:09.796195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00062fc4-d41f-4cf9-b377-b1325261eb7b None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:18:09.869409 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bf53da-652e-4b97-96de-5edca46392de service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:18:09.869926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bf53da-652e-4b97-96de-5edca46392de service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:09.870207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bf53da-652e-4b97-96de-5edca46392de service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:09.870544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bf53da-652e-4b97-96de-5edca46392de service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:09.872175 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-66bf53da-652e-4b97-96de-5edca46392de service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:09.945005 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:09.992294 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bf53da-652e-4b97-96de-5edca46392de service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:18:09.993771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bf53da-652e-4b97-96de-5edca46392de service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:18:10.012931 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 653/1323] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:18:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:18:12.000492 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e67700-e4f9-4fe4-add7-3ef0626ac44c None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:12.001031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e67700-e4f9-4fe4-add7-3ef0626ac44c None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:12.001031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e67700-e4f9-4fe4-add7-3ef0626ac44c None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:12.002599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 671/1324] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:18:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:18:12.007864 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8774f6f2-3d4f-4433-aabe-42c8e4ef34f9 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:12.008325 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8774f6f2-3d4f-4433-aabe-42c8e4ef34f9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:12.008543 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8774f6f2-3d4f-4433-aabe-42c8e4ef34f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:12.074929 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-8774f6f2-3d4f-4433-aabe-42c8e4ef34f9 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:12.076151 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8774f6f2-3d4f-4433-aabe-42c8e4ef34f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:12.173703 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 654/1325] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:18:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:18:12.454931 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e67700-e4f9-4fe4-add7-3ef0626ac44c None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:18:12.456474 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-60e67700-e4f9-4fe4-add7-3ef0626ac44c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:12.570247 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e67700-e4f9-4fe4-add7-3ef0626ac44c None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:18:12.645201 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9867fe3f-bd39-44d7-8bf4-c6a095c64f22 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:18:12.645812 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867fe3f-bd39-44d7-8bf4-c6a095c64f22 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:12.646091 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867fe3f-bd39-44d7-8bf4-c6a095c64f22 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:12.646362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867fe3f-bd39-44d7-8bf4-c6a095c64f22 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:12.647995 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-9867fe3f-bd39-44d7-8bf4-c6a095c64f22 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:12.664512 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:12.759753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9867fe3f-bd39-44d7-8bf4-c6a095c64f22 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:18:12.761777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9867fe3f-bd39-44d7-8bf4-c6a095c64f22 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:18:12.780012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 672/1326] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:18:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:18:14.790164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbedc69-f0b9-44c9-8b98-74f75093b4b4 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:14.790770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbedc69-f0b9-44c9-8b98-74f75093b4b4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:14.790770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbedc69-f0b9-44c9-8b98-74f75093b4b4 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:14.792058 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 655/1327] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:18:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:18:14.797397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0530e68-7065-4a00-a293-9c4c2e7a472d None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:14.797801 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0530e68-7065-4a00-a293-9c4c2e7a472d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:14.797952 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0530e68-7065-4a00-a293-9c4c2e7a472d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:14.864891 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-b0530e68-7065-4a00-a293-9c4c2e7a472d None None] MFA Rules not processed for user `6a1eb60c46aa4d27b91657cd7f9fa60f`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:14.866148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b0530e68-7065-4a00-a293-9c4c2e7a472d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:14.931047 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:14.963123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 673/1328] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:18:14 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:18:15.252891 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbedc69-f0b9-44c9-8b98-74f75093b4b4 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:18:15.254451 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9dbedc69-f0b9-44c9-8b98-74f75093b4b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:15.287493 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:15.367100 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbedc69-f0b9-44c9-8b98-74f75093b4b4 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:18:15.444287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca98dbaf-55b8-4597-bf41-bb6f774465d9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:18:15.444894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca98dbaf-55b8-4597-bf41-bb6f774465d9 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:15.445205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca98dbaf-55b8-4597-bf41-bb6f774465d9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:15.445471 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca98dbaf-55b8-4597-bf41-bb6f774465d9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:15.447247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ca98dbaf-55b8-4597-bf41-bb6f774465d9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:15.526859 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:15.559058 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca98dbaf-55b8-4597-bf41-bb6f774465d9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:18:15.560513 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca98dbaf-55b8-4597-bf41-bb6f774465d9 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:18:15.580713 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 656/1329] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:18:15 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:18:17.739596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf7d3d7-145d-4287-90ce-4990733a16aa None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:17.740010 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf7d3d7-145d-4287-90ce-4990733a16aa None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:17.740151 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf7d3d7-145d-4287-90ce-4990733a16aa None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:17.741555 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 674/1330] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:18:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:18:17.747448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670e82be-f0a6-40c9-9b6b-f72e5fef87b4 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:17.747869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670e82be-f0a6-40c9-9b6b-f72e5fef87b4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:17.747985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670e82be-f0a6-40c9-9b6b-f72e5fef87b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:17.814412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-670e82be-f0a6-40c9-9b6b-f72e5fef87b4 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:17.815554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-670e82be-f0a6-40c9-9b6b-f72e5fef87b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:17.912150 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 657/1331] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:18:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:18:18.199446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf7d3d7-145d-4287-90ce-4990733a16aa None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:18:18.201068 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-faf7d3d7-145d-4287-90ce-4990733a16aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:18.304055 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf7d3d7-145d-4287-90ce-4990733a16aa None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:18:18.377529 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b1cfd5-b98d-41dd-a421-a688b35595b4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:18:18.378050 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b1cfd5-b98d-41dd-a421-a688b35595b4 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:18.378324 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b1cfd5-b98d-41dd-a421-a688b35595b4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:18.378590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b1cfd5-b98d-41dd-a421-a688b35595b4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:18.380179 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-a8b1cfd5-b98d-41dd-a421-a688b35595b4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:18.407568 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:18.490615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b1cfd5-b98d-41dd-a421-a688b35595b4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:18:18.492056 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b1cfd5-b98d-41dd-a421-a688b35595b4 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:18:18.510997 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 675/1332] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:18:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:18:20.482090 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab62b42-61d2-4bb9-8937-7486905bb213 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:20.482663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab62b42-61d2-4bb9-8937-7486905bb213 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:20.482663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab62b42-61d2-4bb9-8937-7486905bb213 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:20.483968 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 658/1333] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:18:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:18:20.489123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14f1a33-079a-48de-b4c5-5155ff4bc861 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:20.489549 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14f1a33-079a-48de-b4c5-5155ff4bc861 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:20.489678 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14f1a33-079a-48de-b4c5-5155ff4bc861 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:20.557376 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-f14f1a33-079a-48de-b4c5-5155ff4bc861 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:20.558583 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f14f1a33-079a-48de-b4c5-5155ff4bc861 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:20.633720 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:20.657270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 676/1334] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:18:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:18:20.942112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab62b42-61d2-4bb9-8937-7486905bb213 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:18:20.943674 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5ab62b42-61d2-4bb9-8937-7486905bb213 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:20.984755 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:21.059179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab62b42-61d2-4bb9-8937-7486905bb213 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:18:21.132088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b275c3-0d2a-44f9-8654-4c93e7063cce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:18:21.132696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b275c3-0d2a-44f9-8654-4c93e7063cce service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:21.132977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b275c3-0d2a-44f9-8654-4c93e7063cce service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:21.133279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b275c3-0d2a-44f9-8654-4c93e7063cce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:21.134879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c4b275c3-0d2a-44f9-8654-4c93e7063cce service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:21.217193 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:21.242782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b275c3-0d2a-44f9-8654-4c93e7063cce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:18:21.244280 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b275c3-0d2a-44f9-8654-4c93e7063cce service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:18:21.263030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 659/1335] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:18:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:18:23.481559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe6de02-8932-4cba-a777-2307c79d87d8 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:23.482104 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe6de02-8932-4cba-a777-2307c79d87d8 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:23.482251 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe6de02-8932-4cba-a777-2307c79d87d8 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:23.483635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 677/1336] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:18:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:18:23.488798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a95eae-e1b9-4440-9aa8-2fe368798f7e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:23.489205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a95eae-e1b9-4440-9aa8-2fe368798f7e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:23.489376 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a95eae-e1b9-4440-9aa8-2fe368798f7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:23.556187 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-77a95eae-e1b9-4440-9aa8-2fe368798f7e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:23.557922 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-77a95eae-e1b9-4440-9aa8-2fe368798f7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:23.653641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 660/1337] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:18:23 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:18:23.935583 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe6de02-8932-4cba-a777-2307c79d87d8 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:18:23.937174 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-fbe6de02-8932-4cba-a777-2307c79d87d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:24.043836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe6de02-8932-4cba-a777-2307c79d87d8 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:18:24.116062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bbb792-e900-4af5-a81c-ff4c231c211e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:18:24.116681 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bbb792-e900-4af5-a81c-ff4c231c211e service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:24.116961 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bbb792-e900-4af5-a81c-ff4c231c211e service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:24.117264 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bbb792-e900-4af5-a81c-ff4c231c211e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:24.119008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-12bbb792-e900-4af5-a81c-ff4c231c211e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:24.162909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:24.236300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bbb792-e900-4af5-a81c-ff4c231c211e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:18:24.238037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bbb792-e900-4af5-a81c-ff4c231c211e service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:18:24.256579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 678/1338] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:18:23 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:18:28.075715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be75372-60ae-4354-bff0-e95743dfec9e None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:28.076267 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be75372-60ae-4354-bff0-e95743dfec9e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:28.076267 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be75372-60ae-4354-bff0-e95743dfec9e None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:28.077638 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 661/1339] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:18:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:18:28.082904 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a109242-0827-4893-a73e-a6f91fce37c9 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:28.083339 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a109242-0827-4893-a73e-a6f91fce37c9 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:28.083439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a109242-0827-4893-a73e-a6f91fce37c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:28.153447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-4a109242-0827-4893-a73e-a6f91fce37c9 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:28.154660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4a109242-0827-4893-a73e-a6f91fce37c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:28.244437 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:28.257937 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 679/1340] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:18:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:18:28.539262 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be75372-60ae-4354-bff0-e95743dfec9e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:18:28.540798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7be75372-60ae-4354-bff0-e95743dfec9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:28.591517 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:28.652404 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be75372-60ae-4354-bff0-e95743dfec9e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:18:28.727904 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa262ce-db15-462b-a89a-6d0a30d534d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:18:28.728502 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa262ce-db15-462b-a89a-6d0a30d534d0 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:28.728777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa262ce-db15-462b-a89a-6d0a30d534d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:28.729132 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa262ce-db15-462b-a89a-6d0a30d534d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:28.730712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7aa262ce-db15-462b-a89a-6d0a30d534d0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:28.824054 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:28.840344 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa262ce-db15-462b-a89a-6d0a30d534d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:18:28.841804 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa262ce-db15-462b-a89a-6d0a30d534d0 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:18:28.859546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 662/1341] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:18:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:18:31.516181 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c065fb3d-dee7-4211-bfd5-4c9f79f24443 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:31.516734 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c065fb3d-dee7-4211-bfd5-4c9f79f24443 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:31.516734 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c065fb3d-dee7-4211-bfd5-4c9f79f24443 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:31.518102 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 680/1342] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:18:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:18:31.523602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc9d92a-85a8-43b2-a70b-c8c4c42e7b06 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:31.524170 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc9d92a-85a8-43b2-a70b-c8c4c42e7b06 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:31.524170 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc9d92a-85a8-43b2-a70b-c8c4c42e7b06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:31.592024 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-dfc9d92a-85a8-43b2-a70b-c8c4c42e7b06 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:31.593692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-dfc9d92a-85a8-43b2-a70b-c8c4c42e7b06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:31.699104 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 663/1343] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:18:31 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:18:31.982232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c065fb3d-dee7-4211-bfd5-4c9f79f24443 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:18:31.983904 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c065fb3d-dee7-4211-bfd5-4c9f79f24443 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:32.092005 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c065fb3d-dee7-4211-bfd5-4c9f79f24443 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:18:32.171888 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f162487-bede-4da1-b02c-2beb3c6f0122 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:18:32.172493 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f162487-bede-4da1-b02c-2beb3c6f0122 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:32.172773 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f162487-bede-4da1-b02c-2beb3c6f0122 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:32.173071 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f162487-bede-4da1-b02c-2beb3c6f0122 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:32.174658 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-1f162487-bede-4da1-b02c-2beb3c6f0122 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:32.217915 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:32.286961 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f162487-bede-4da1-b02c-2beb3c6f0122 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:18:32.288428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f162487-bede-4da1-b02c-2beb3c6f0122 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:18:32.308425 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 681/1344] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:18:31 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:18:36.210087 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6e31db-efb1-4f7b-b98e-3dd38138e9ff None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:36.210541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6e31db-efb1-4f7b-b98e-3dd38138e9ff None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:36.210663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6e31db-efb1-4f7b-b98e-3dd38138e9ff None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:36.212143 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 664/1345] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:18:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:18:36.218049 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da5be41-8ff3-484c-a5e2-b053b59067af None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:36.218479 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da5be41-8ff3-484c-a5e2-b053b59067af None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:36.218601 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da5be41-8ff3-484c-a5e2-b053b59067af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:36.294760 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-4da5be41-8ff3-484c-a5e2-b053b59067af None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:36.296044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4da5be41-8ff3-484c-a5e2-b053b59067af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:36.400072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:36.403610 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 682/1346] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:18:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:18:36.702244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e78cc7-432e-4659-8d26-7488c9a96490 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:36.702603 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e78cc7-432e-4659-8d26-7488c9a96490 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:36.702965 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e78cc7-432e-4659-8d26-7488c9a96490 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:36.704448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 665/1347] 10.222.0.85 () {54 vars in 965 bytes} [Tue Aug 30 04:18:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:18:36.711004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318f7902-1c47-4289-9d72-a1817bc1b9cb None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:36.711291 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318f7902-1c47-4289-9d72-a1817bc1b9cb None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:36.711568 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318f7902-1c47-4289-9d72-a1817bc1b9cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:36.790469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-318f7902-1c47-4289-9d72-a1817bc1b9cb None None] MFA Rules not processed for user `5c2edd77896a4534888d6758ec0ee0bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:36.791685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-318f7902-1c47-4289-9d72-a1817bc1b9cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:36.869840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 683/1348] 10.222.0.85 () {58 vars in 1062 bytes} [Tue Aug 30 04:18:36 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:18:36.878888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e78cc7-432e-4659-8d26-7488c9a96490 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:18:36.880426 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f7e78cc7-432e-4659-8d26-7488c9a96490 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:36.937736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:36.983173 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e78cc7-432e-4659-8d26-7488c9a96490 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:18:37.056657 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490aa420-6a24-4c6d-b2f2-35b173160ad6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:18:37.057320 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490aa420-6a24-4c6d-b2f2-35b173160ad6 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:37.057596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490aa420-6a24-4c6d-b2f2-35b173160ad6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:37.057862 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490aa420-6a24-4c6d-b2f2-35b173160ad6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:37.059446 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-490aa420-6a24-4c6d-b2f2-35b173160ad6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:37.166066 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:37.170991 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490aa420-6a24-4c6d-b2f2-35b173160ad6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:18:37.172515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490aa420-6a24-4c6d-b2f2-35b173160ad6 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:18:37.191466 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 666/1349] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:18:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:18:42.584180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aed55d0-b2d1-4ea2-85a1-f5ead48279d2 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:42.584604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aed55d0-b2d1-4ea2-85a1-f5ead48279d2 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:42.584703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aed55d0-b2d1-4ea2-85a1-f5ead48279d2 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:42.586785 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 684/1350] 10.222.0.85 () {54 vars in 921 bytes} [Tue Aug 30 04:18:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:18:42.592089 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f2991c-1026-4e57-9ce1-684d2f50b48b None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:42.592551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f2991c-1026-4e57-9ce1-684d2f50b48b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:42.592742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f2991c-1026-4e57-9ce1-684d2f50b48b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:42.694549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-06f2991c-1026-4e57-9ce1-684d2f50b48b None None] MFA Rules not processed for user `359d79cbfcad49beba2b495066824128`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:42.695784 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-06f2991c-1026-4e57-9ce1-684d2f50b48b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:42.829602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 667/1351] 10.222.0.85 () {58 vars in 1018 bytes} [Tue Aug 30 04:18:42 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:18:46.867387 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=59 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:47.151152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67723d3-cabd-4c19-915e-46faadbe7ba1 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:47.151352 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67723d3-cabd-4c19-915e-46faadbe7ba1 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:47.151690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67723d3-cabd-4c19-915e-46faadbe7ba1 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:47.153195 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 685/1352] 10.222.0.85 () {54 vars in 921 bytes} [Tue Aug 30 04:18:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:18:47.158747 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380c97fb-232e-43f3-9547-43d128c27afc None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:47.159235 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380c97fb-232e-43f3-9547-43d128c27afc None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:47.159486 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380c97fb-232e-43f3-9547-43d128c27afc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:47.216938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:47.243865 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-380c97fb-232e-43f3-9547-43d128c27afc None None] MFA Rules not processed for user `359d79cbfcad49beba2b495066824128`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:47.245823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-380c97fb-232e-43f3-9547-43d128c27afc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:47.366091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 668/1353] 10.222.0.85 () {58 vars in 1018 bytes} [Tue Aug 30 04:18:47 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:18:48.747053 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbd9f98-b3b2-4e3f-8473-a51b2a0e0dbc None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:48.747638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbd9f98-b3b2-4e3f-8473-a51b2a0e0dbc None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:48.747744 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbd9f98-b3b2-4e3f-8473-a51b2a0e0dbc None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:48.749443 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 686/1354] 10.222.0.85 () {54 vars in 921 bytes} [Tue Aug 30 04:18:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:18:48.754466 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f6a63e-8531-4c54-af66-4207eff08c34 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:48.754900 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f6a63e-8531-4c54-af66-4207eff08c34 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:48.755019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f6a63e-8531-4c54-af66-4207eff08c34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:48.836169 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-38f6a63e-8531-4c54-af66-4207eff08c34 None None] MFA Rules not processed for user `359d79cbfcad49beba2b495066824128`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:48.837442 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-38f6a63e-8531-4c54-af66-4207eff08c34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:48.881389 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:48.949496 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 669/1355] 10.222.0.85 () {58 vars in 1018 bytes} [Tue Aug 30 04:18:48 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:18:52.786440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba6d958-6d64-4f0e-835b-a96196e34994 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:52.787033 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba6d958-6d64-4f0e-835b-a96196e34994 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:52.787033 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba6d958-6d64-4f0e-835b-a96196e34994 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:52.788442 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 687/1356] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:18:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:18:52.794499 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afedeeec-c9e8-4fef-898b-c6ab3bc27698 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:52.794955 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afedeeec-c9e8-4fef-898b-c6ab3bc27698 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:52.795053 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afedeeec-c9e8-4fef-898b-c6ab3bc27698 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:52.880381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-afedeeec-c9e8-4fef-898b-c6ab3bc27698 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:52.889854 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-afedeeec-c9e8-4fef-898b-c6ab3bc27698 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:53.039250 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 670/1357] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:18:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:18:53.340366 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a5ca92-5eb4-44ec-a18b-f5743103ba48 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:53.340982 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a5ca92-5eb4-44ec-a18b-f5743103ba48 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:53.341553 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a5ca92-5eb4-44ec-a18b-f5743103ba48 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:53.343468 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 688/1358] 10.222.0.85 () {54 vars in 962 bytes} [Tue Aug 30 04:18:53 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:18:53.352930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cc00ae-b200-4728-b823-6d8a66f36056 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:53.354182 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cc00ae-b200-4728-b823-6d8a66f36056 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:53.354823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cc00ae-b200-4728-b823-6d8a66f36056 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:53.377838 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:53.443647 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-96cc00ae-b200-4728-b823-6d8a66f36056 None None] MFA Rules not processed for user `4bc4d455ebf041d488b1199ca58358a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:53.444890 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-96cc00ae-b200-4728-b823-6d8a66f36056 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:53.566591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 671/1359] 10.222.0.85 () {58 vars in 1059 bytes} [Tue Aug 30 04:18:53 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:18:53.574358 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a5ca92-5eb4-44ec-a18b-f5743103ba48 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:18:53.575849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-40a5ca92-5eb4-44ec-a18b-f5743103ba48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:53.724002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a5ca92-5eb4-44ec-a18b-f5743103ba48 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:18:53.801283 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:53.843838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4bca79-6030-4329-a353-3a7a0ff19378 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:18:53.844346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4bca79-6030-4329-a353-3a7a0ff19378 service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:53.844619 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4bca79-6030-4329-a353-3a7a0ff19378 service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:53.844883 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4bca79-6030-4329-a353-3a7a0ff19378 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:53.846496 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-9e4bca79-6030-4329-a353-3a7a0ff19378 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:53.939191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4bca79-6030-4329-a353-3a7a0ff19378 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:18:53.940792 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4bca79-6030-4329-a353-3a7a0ff19378 service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:18:53.959978 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 689/1360] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:18:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:18:56.105990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe632b19-e7b4-4c57-908f-15cf5ed829a2 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:56.106430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe632b19-e7b4-4c57-908f-15cf5ed829a2 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:56.106582 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe632b19-e7b4-4c57-908f-15cf5ed829a2 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:56.111938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 672/1361] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:18:56 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:18:56.113779 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adcf00a-d8a9-46d8-a50a-9f1e4bfa9ddd None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:56.115348 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adcf00a-d8a9-46d8-a50a-9f1e4bfa9ddd None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:56.115628 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adcf00a-d8a9-46d8-a50a-9f1e4bfa9ddd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:56.189301 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-3adcf00a-d8a9-46d8-a50a-9f1e4bfa9ddd None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:56.190526 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-3adcf00a-d8a9-46d8-a50a-9f1e4bfa9ddd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:56.231388 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:56.305666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 690/1362] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:18:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:18:56.596005 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe632b19-e7b4-4c57-908f-15cf5ed829a2 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:18:56.598651 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fe632b19-e7b4-4c57-908f-15cf5ed829a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:56.677883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:56.782897 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe632b19-e7b4-4c57-908f-15cf5ed829a2 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:18:56.928980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748b0661-6f07-4367-913f-0447995b8b87 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:18:56.929747 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b0661-6f07-4367-913f-0447995b8b87 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:56.929899 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b0661-6f07-4367-913f-0447995b8b87 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:56.930143 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b0661-6f07-4367-913f-0447995b8b87 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:56.931828 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-748b0661-6f07-4367-913f-0447995b8b87 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:56.956393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:57.028618 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748b0661-6f07-4367-913f-0447995b8b87 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:18:57.030176 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748b0661-6f07-4367-913f-0447995b8b87 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:18:57.049169 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 673/1363] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:18:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 454 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:18:59.319620 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d9fcb3-fc96-428f-bbff-99684ad30c58 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:59.320626 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d9fcb3-fc96-428f-bbff-99684ad30c58 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:59.321002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d9fcb3-fc96-428f-bbff-99684ad30c58 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:59.322837 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 691/1364] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:18:59 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:18:59.328728 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a300bc-fef7-4b52-bac6-96bfbaa9be6a None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:59.329266 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a300bc-fef7-4b52-bac6-96bfbaa9be6a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:59.329345 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a300bc-fef7-4b52-bac6-96bfbaa9be6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:59.409738 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-63a300bc-fef7-4b52-bac6-96bfbaa9be6a None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:59.410919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-63a300bc-fef7-4b52-bac6-96bfbaa9be6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:18:59.528679 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:18:59.533159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 674/1365] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:18:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:18:59.836470 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db50620-586b-4d71-b198-83c2dcc93a7c None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:59.836843 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db50620-586b-4d71-b198-83c2dcc93a7c None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:59.837649 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db50620-586b-4d71-b198-83c2dcc93a7c None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:59.839052 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 692/1366] 10.222.0.85 () {54 vars in 962 bytes} [Tue Aug 30 04:18:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:18:59.845195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83762bfe-3230-492e-822b-1797333acd0a None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:18:59.845480 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83762bfe-3230-492e-822b-1797333acd0a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:18:59.845750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83762bfe-3230-492e-822b-1797333acd0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:18:59.935102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-83762bfe-3230-492e-822b-1797333acd0a None None] MFA Rules not processed for user `4bc4d455ebf041d488b1199ca58358a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:18:59.936441 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-83762bfe-3230-492e-822b-1797333acd0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:00.083927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 675/1367] 10.222.0.85 () {58 vars in 1059 bytes} [Tue Aug 30 04:18:59 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:00.093194 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db50620-586b-4d71-b198-83c2dcc93a7c None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:00.094716 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-7db50620-586b-4d71-b198-83c2dcc93a7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:00.249964 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db50620-586b-4d71-b198-83c2dcc93a7c None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:00.271725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:00.378274 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d638d1-c0a8-4b68-847b-117321849cb0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:00.378912 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d638d1-c0a8-4b68-847b-117321849cb0 service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:00.379095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d638d1-c0a8-4b68-847b-117321849cb0 service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:00.379363 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d638d1-c0a8-4b68-847b-117321849cb0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:00.381013 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-47d638d1-c0a8-4b68-847b-117321849cb0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:00.482527 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d638d1-c0a8-4b68-847b-117321849cb0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:00.484403 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d638d1-c0a8-4b68-847b-117321849cb0 service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:00.507275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 693/1368] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:19:01.286093 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a4fc92-12cd-46a5-8b1f-b9e408399534 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:01.286648 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a4fc92-12cd-46a5-8b1f-b9e408399534 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:01.286648 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a4fc92-12cd-46a5-8b1f-b9e408399534 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:01.288295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 676/1369] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:19:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:01.293370 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7680bd9-1254-4db6-9f5a-1a4b3bb35085 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:01.293661 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7680bd9-1254-4db6-9f5a-1a4b3bb35085 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:01.293935 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7680bd9-1254-4db6-9f5a-1a4b3bb35085 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:01.328454 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:01.367776 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-d7680bd9-1254-4db6-9f5a-1a4b3bb35085 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:01.369531 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d7680bd9-1254-4db6-9f5a-1a4b3bb35085 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:01.484910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 694/1370] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:19:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:02.094201 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4a044a-90e1-4ce4-bf1b-13b2bba74720 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:02.094635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4a044a-90e1-4ce4-bf1b-13b2bba74720 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:02.095294 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4a044a-90e1-4ce4-bf1b-13b2bba74720 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:02.096731 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 677/1371] 10.222.0.85 () {54 vars in 964 bytes} [Tue Aug 30 04:19:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:02.101716 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec83703-5c1c-4fee-94fa-71b31b003d11 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:02.102129 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec83703-5c1c-4fee-94fa-71b31b003d11 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:02.102514 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec83703-5c1c-4fee-94fa-71b31b003d11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:02.190666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-cec83703-5c1c-4fee-94fa-71b31b003d11 None None] MFA Rules not processed for user `7ef280ce39b14085be329f60b22530d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:02.192052 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-cec83703-5c1c-4fee-94fa-71b31b003d11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:02.247182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:02.280911 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 695/1372] 10.222.0.85 () {58 vars in 1061 bytes} [Tue Aug 30 04:19:02 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:02.289838 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4a044a-90e1-4ce4-bf1b-13b2bba74720 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:02.296685 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0a4a044a-90e1-4ce4-bf1b-13b2bba74720 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:02.358551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:02.421592 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4a044a-90e1-4ce4-bf1b-13b2bba74720 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:02.513248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2dc4cfb-1a04-480e-8ee9-c92c5b056cbe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:02.514024 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2dc4cfb-1a04-480e-8ee9-c92c5b056cbe service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:02.514432 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2dc4cfb-1a04-480e-8ee9-c92c5b056cbe service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:02.514809 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2dc4cfb-1a04-480e-8ee9-c92c5b056cbe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:02.516816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b2dc4cfb-1a04-480e-8ee9-c92c5b056cbe service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:02.534745 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd650ff6-e27c-4f78-bcf8-abaa99f0647d None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:02.535670 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd650ff6-e27c-4f78-bcf8-abaa99f0647d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:02.535952 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd650ff6-e27c-4f78-bcf8-abaa99f0647d None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:02.542443 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 696/1373] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:19:02 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:19:02.543739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7201208-7ba1-4602-8101-289fe7c0e869 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:02.544150 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7201208-7ba1-4602-8101-289fe7c0e869 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:02.544285 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7201208-7ba1-4602-8101-289fe7c0e869 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:02.634495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-c7201208-7ba1-4602-8101-289fe7c0e869 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:02.635788 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c7201208-7ba1-4602-8101-289fe7c0e869 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:02.646737 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:02.666082 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2dc4cfb-1a04-480e-8ee9-c92c5b056cbe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:02.668328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2dc4cfb-1a04-480e-8ee9-c92c5b056cbe service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:02.701907 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 678/1374] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:19:02.771886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 697/1375] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:19:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:03.060052 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2dc4cfb-1a04-480e-8ee9-c92c5b056cbe service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:03.061293 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b2dc4cfb-1a04-480e-8ee9-c92c5b056cbe service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:03.166180 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2dc4cfb-1a04-480e-8ee9-c92c5b056cbe service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:03.270342 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:03.283081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3592960-4c65-4d58-bf6d-e92b77b95673 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:03.283716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3592960-4c65-4d58-bf6d-e92b77b95673 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:03.283993 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3592960-4c65-4d58-bf6d-e92b77b95673 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:03.284266 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3592960-4c65-4d58-bf6d-e92b77b95673 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:03.285903 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f3592960-4c65-4d58-bf6d-e92b77b95673 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:03.377171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3592960-4c65-4d58-bf6d-e92b77b95673 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:03.378714 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3592960-4c65-4d58-bf6d-e92b77b95673 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:03.397132 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 679/1376] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:19:05.364754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ee710e-cd5a-4995-84c9-0155a33fce4f None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:05.365612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ee710e-cd5a-4995-84c9-0155a33fce4f None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:05.365902 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ee710e-cd5a-4995-84c9-0155a33fce4f None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:05.367235 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 698/1377] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:19:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:05.372632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0b4fd9-bc5c-4b85-a515-d39f9a759a9a None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:05.377156 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0b4fd9-bc5c-4b85-a515-d39f9a759a9a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:05.377452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0b4fd9-bc5c-4b85-a515-d39f9a759a9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:05.459450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-2e0b4fd9-bc5c-4b85-a515-d39f9a759a9a None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:05.460881 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2e0b4fd9-bc5c-4b85-a515-d39f9a759a9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:05.523313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:05.566542 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 680/1378] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:19:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:05.856438 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ee710e-cd5a-4995-84c9-0155a33fce4f None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:05.858152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-75ee710e-cd5a-4995-84c9-0155a33fce4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:05.927210 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:06.024598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ee710e-cd5a-4995-84c9-0155a33fce4f None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:06.144271 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc6b0b3-c615-4f1b-aee4-3e11e31efd33 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:06.145059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc6b0b3-c615-4f1b-aee4-3e11e31efd33 service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:06.145343 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc6b0b3-c615-4f1b-aee4-3e11e31efd33 service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:06.145627 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc6b0b3-c615-4f1b-aee4-3e11e31efd33 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:06.147245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-bfc6b0b3-c615-4f1b-aee4-3e11e31efd33 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:06.162173 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:06.264829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc6b0b3-c615-4f1b-aee4-3e11e31efd33 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:06.266500 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc6b0b3-c615-4f1b-aee4-3e11e31efd33 service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:06.285573 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 699/1379] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:19:08.318030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc268098-dcce-4dee-8a4c-e83f8fd23662 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:08.318555 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc268098-dcce-4dee-8a4c-e83f8fd23662 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:08.318555 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc268098-dcce-4dee-8a4c-e83f8fd23662 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:08.320141 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 681/1380] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:19:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:08.325417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc3e985-98c6-4a48-9b4a-13e5f09b51ce None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:08.325850 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc3e985-98c6-4a48-9b4a-13e5f09b51ce None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:08.325955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc3e985-98c6-4a48-9b4a-13e5f09b51ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:08.399689 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-6fc3e985-98c6-4a48-9b4a-13e5f09b51ce None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:08.400904 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-6fc3e985-98c6-4a48-9b4a-13e5f09b51ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:08.505792 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:08.519437 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 700/1381] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:19:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:08.809136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc268098-dcce-4dee-8a4c-e83f8fd23662 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:08.810816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cc268098-dcce-4dee-8a4c-e83f8fd23662 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:08.990392 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc268098-dcce-4dee-8a4c-e83f8fd23662 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:09.043995 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:09.125989 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7eb6005-c57f-4d45-b726-1a4d19c42e79 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:09.126963 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7eb6005-c57f-4d45-b726-1a4d19c42e79 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:09.127346 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7eb6005-c57f-4d45-b726-1a4d19c42e79 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:09.127720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7eb6005-c57f-4d45-b726-1a4d19c42e79 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:09.129801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d7eb6005-c57f-4d45-b726-1a4d19c42e79 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:09.236884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7eb6005-c57f-4d45-b726-1a4d19c42e79 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:09.238752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7eb6005-c57f-4d45-b726-1a4d19c42e79 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:09.267743 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 682/1382] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 459 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:19:11.299259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d89727-95c6-45b3-ba85-4e2cd7617bc4 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:11.299850 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d89727-95c6-45b3-ba85-4e2cd7617bc4 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:11.299955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d89727-95c6-45b3-ba85-4e2cd7617bc4 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:11.301815 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 701/1383] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:19:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:11.307985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcb7541-5fc2-4139-b5c2-d696e6161460 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:11.308557 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcb7541-5fc2-4139-b5c2-d696e6161460 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:11.308557 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcb7541-5fc2-4139-b5c2-d696e6161460 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:11.378464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-0bcb7541-5fc2-4139-b5c2-d696e6161460 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:11.379621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0bcb7541-5fc2-4139-b5c2-d696e6161460 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:11.381721 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:11.497301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 683/1384] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:19:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:11.795472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d89727-95c6-45b3-ba85-4e2cd7617bc4 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:11.797128 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b4d89727-95c6-45b3-ba85-4e2cd7617bc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:11.956984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d89727-95c6-45b3-ba85-4e2cd7617bc4 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:12.030912 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:12.091069 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623a238a-5ad9-480d-8fcd-181692d6cba2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:12.091681 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623a238a-5ad9-480d-8fcd-181692d6cba2 service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:12.091962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623a238a-5ad9-480d-8fcd-181692d6cba2 service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:12.092234 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623a238a-5ad9-480d-8fcd-181692d6cba2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:12.093896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-623a238a-5ad9-480d-8fcd-181692d6cba2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:12.200564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623a238a-5ad9-480d-8fcd-181692d6cba2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:12.202194 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623a238a-5ad9-480d-8fcd-181692d6cba2 service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:12.222680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 702/1385] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:19:14.167157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804ad80a-e28c-4779-ac18-cb77f8b2d7af None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:14.167720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804ad80a-e28c-4779-ac18-cb77f8b2d7af None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:14.168103 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804ad80a-e28c-4779-ac18-cb77f8b2d7af None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:14.169580 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 684/1386] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:19:14 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:14.174916 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f07660c-8ddc-41c4-9c97-fa19d98fd9f7 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:14.175494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f07660c-8ddc-41c4-9c97-fa19d98fd9f7 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:14.175494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f07660c-8ddc-41c4-9c97-fa19d98fd9f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:14.250356 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-8f07660c-8ddc-41c4-9c97-fa19d98fd9f7 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:14.251562 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8f07660c-8ddc-41c4-9c97-fa19d98fd9f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:14.279811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:14.371955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 703/1387] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:19:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:19:14.671158 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804ad80a-e28c-4779-ac18-cb77f8b2d7af None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:14.673730 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-804ad80a-e28c-4779-ac18-cb77f8b2d7af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:14.835037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:14.859827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804ad80a-e28c-4779-ac18-cb77f8b2d7af None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:14.986833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c031b240-07b5-43a2-9568-a0f15119738d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:14.987450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c031b240-07b5-43a2-9568-a0f15119738d service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:14.987729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c031b240-07b5-43a2-9568-a0f15119738d service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:14.988001 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c031b240-07b5-43a2-9568-a0f15119738d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:14.989671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c031b240-07b5-43a2-9568-a0f15119738d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:15.095414 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c031b240-07b5-43a2-9568-a0f15119738d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:15.097012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c031b240-07b5-43a2-9568-a0f15119738d service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:15.103078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:15.123207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 685/1388] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:19:17.328440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3433f13c-ab08-4007-9bf9-ab9b47fbf837 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:17.329010 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3433f13c-ab08-4007-9bf9-ab9b47fbf837 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:17.329010 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3433f13c-ab08-4007-9bf9-ab9b47fbf837 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:17.330371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 704/1389] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:19:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:17.335641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3242c815-57ec-4519-9aad-44ba9115bf21 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:17.336091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3242c815-57ec-4519-9aad-44ba9115bf21 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:17.336325 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3242c815-57ec-4519-9aad-44ba9115bf21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:17.405524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3242c815-57ec-4519-9aad-44ba9115bf21 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:17.406643 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3242c815-57ec-4519-9aad-44ba9115bf21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:17.507482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 686/1390] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:19:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:17.818098 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3433f13c-ab08-4007-9bf9-ab9b47fbf837 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:17.819649 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-3433f13c-ab08-4007-9bf9-ab9b47fbf837 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:17.973862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3433f13c-ab08-4007-9bf9-ab9b47fbf837 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:17.984554 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:18.095002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00df7861-6bbe-4fd9-99c5-edabf44ad0d2 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:18.095209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00df7861-6bbe-4fd9-99c5-edabf44ad0d2 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:18.095455 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00df7861-6bbe-4fd9-99c5-edabf44ad0d2 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:18.096816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 687/1391] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:19:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:19:18.102092 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61de3bc8-58cb-4c65-8a7a-17689745eeb8 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:18.102370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61de3bc8-58cb-4c65-8a7a-17689745eeb8 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:18.102650 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61de3bc8-58cb-4c65-8a7a-17689745eeb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:18.104248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e6971c-47f3-4204-a494-c46c18a88e28 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:18.104858 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e6971c-47f3-4204-a494-c46c18a88e28 service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:18.105164 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e6971c-47f3-4204-a494-c46c18a88e28 service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:18.105438 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e6971c-47f3-4204-a494-c46c18a88e28 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:18.107114 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-a8e6971c-47f3-4204-a494-c46c18a88e28 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:18.182536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-61de3bc8-58cb-4c65-8a7a-17689745eeb8 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:18.183845 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-61de3bc8-58cb-4c65-8a7a-17689745eeb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:18.206038 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:18.237303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e6971c-47f3-4204-a494-c46c18a88e28 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:18.240384 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e6971c-47f3-4204-a494-c46c18a88e28 service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:18.271909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 705/1392] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 454 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:19:18.294443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 688/1393] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:19:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:19:18.599727 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa58860-5b36-41b4-81f4-106053ccb87d None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:18.600280 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa58860-5b36-41b4-81f4-106053ccb87d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:18.600452 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa58860-5b36-41b4-81f4-106053ccb87d None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:18.601937 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 706/1394] 10.222.0.85 () {54 vars in 967 bytes} [Tue Aug 30 04:19:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:18.607298 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd797e2-68aa-4f9c-b7e5-67527b6484c0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:18.607750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd797e2-68aa-4f9c-b7e5-67527b6484c0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:18.607855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd797e2-68aa-4f9c-b7e5-67527b6484c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:18.689465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4fd797e2-68aa-4f9c-b7e5-67527b6484c0 None None] MFA Rules not processed for user `354b8228661c4369b3bca2e631f3f140`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:18.690838 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4fd797e2-68aa-4f9c-b7e5-67527b6484c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:18.778351 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 689/1395] 10.222.0.85 () {58 vars in 1064 bytes} [Tue Aug 30 04:19:18 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:18.785919 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa58860-5b36-41b4-81f4-106053ccb87d None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:18.787625 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-cfa58860-5b36-41b4-81f4-106053ccb87d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:18.801486 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:18.898345 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa58860-5b36-41b4-81f4-106053ccb87d None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:18.983317 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db35861f-14d4-43a5-957f-5fa179ca667e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '354b8228661c4369b3bca2e631f3f140', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:18.983907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db35861f-14d4-43a5-957f-5fa179ca667e service glance] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:18.984182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db35861f-14d4-43a5-957f-5fa179ca667e service glance] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:18.984452 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db35861f-14d4-43a5-957f-5fa179ca667e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:18.986070 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-db35861f-14d4-43a5-957f-5fa179ca667e service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:19.059513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:19.113864 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db35861f-14d4-43a5-957f-5fa179ca667e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:19.115894 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db35861f-14d4-43a5-957f-5fa179ca667e service glance] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:19.138054 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 707/1396] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:19:19.908924 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f2649b-5471-4e6b-ab2f-9aaab208c5ff None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:19.909487 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f2649b-5471-4e6b-ab2f-9aaab208c5ff None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:19.909487 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f2649b-5471-4e6b-ab2f-9aaab208c5ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:19.930857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:19.981088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-45f2649b-5471-4e6b-ab2f-9aaab208c5ff None None] MFA Rules not processed for user `00aeab772672403984bdcb8baaa1520e`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:19.982374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-45f2649b-5471-4e6b-ab2f-9aaab208c5ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:20.088321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 690/1397] 10.222.0.85 () {58 vars in 1009 bytes} [Tue Aug 30 04:19:19 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:20.135256 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0403d55f-8e43-4a5c-9688-a6574b70528c None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:20.135739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0403d55f-8e43-4a5c-9688-a6574b70528c None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:20.135848 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0403d55f-8e43-4a5c-9688-a6574b70528c None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:20.137175 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 708/1398] 10.222.0.85 () {54 vars in 943 bytes} [Tue Aug 30 04:19:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:20.144086 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c91ead2-d467-495a-b48b-5b051d138487 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:20.144372 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c91ead2-d467-495a-b48b-5b051d138487 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:20.144690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c91ead2-d467-495a-b48b-5b051d138487 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:20.225513 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-9c91ead2-d467-495a-b48b-5b051d138487 None None] MFA Rules not processed for user `6cb539ef6a69406baa70f660e1a815b4`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:20.227071 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9c91ead2-d467-495a-b48b-5b051d138487 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:20.279578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:20.301002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e971c167-8ac0-4c03-bd23-8fa6811d511c None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:20.301308 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e971c167-8ac0-4c03-bd23-8fa6811d511c None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:20.301577 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e971c167-8ac0-4c03-bd23-8fa6811d511c None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:20.303035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 709/1399] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:19:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:19:20.309370 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feda0f0e-8f18-43ff-9b53-e7060ded68f9 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:20.309656 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feda0f0e-8f18-43ff-9b53-e7060ded68f9 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:20.309928 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feda0f0e-8f18-43ff-9b53-e7060ded68f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:20.354655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 691/1400] 10.222.0.85 () {58 vars in 1040 bytes} [Tue Aug 30 04:19:20 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:20.364947 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c91ead2-d467-495a-b48b-5b051d138487 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:20.376915 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c91ead2-d467-495a-b48b-5b051d138487 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:20.385716 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-feda0f0e-8f18-43ff-9b53-e7060ded68f9 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:20.386950 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-feda0f0e-8f18-43ff-9b53-e7060ded68f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:20.488282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 710/1401] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:19:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:19:20.490154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85c8ba9-aa5f-4a10-8575-63eb3f3a830e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cb539ef6a69406baa70f660e1a815b4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:20.490771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85c8ba9-aa5f-4a10-8575-63eb3f3a830e service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:20.491053 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85c8ba9-aa5f-4a10-8575-63eb3f3a830e service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:20.491324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85c8ba9-aa5f-4a10-8575-63eb3f3a830e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:20.493629 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b85c8ba9-aa5f-4a10-8575-63eb3f3a830e service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:20.549646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:20.582624 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85c8ba9-aa5f-4a10-8575-63eb3f3a830e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:20.584430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85c8ba9-aa5f-4a10-8575-63eb3f3a830e service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:20.593545 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 692/1402] 10.222.0.85 () {60 vars in 1384 bytes} [Tue Aug 30 04:19:20 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 30 04:19:20.797952 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feda0f0e-8f18-43ff-9b53-e7060ded68f9 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:20.801077 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-feda0f0e-8f18-43ff-9b53-e7060ded68f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:20.828861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:20.957770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feda0f0e-8f18-43ff-9b53-e7060ded68f9 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:21.094901 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02fc126-436b-49be-86c9-7d7f3421ee6e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:21.095875 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02fc126-436b-49be-86c9-7d7f3421ee6e service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:21.096182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02fc126-436b-49be-86c9-7d7f3421ee6e service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:21.096456 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02fc126-436b-49be-86c9-7d7f3421ee6e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:21.098442 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e02fc126-436b-49be-86c9-7d7f3421ee6e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:21.127109 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:21.212074 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02fc126-436b-49be-86c9-7d7f3421ee6e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:21.213729 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02fc126-436b-49be-86c9-7d7f3421ee6e service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:21.248555 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 711/1403] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 448 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:19:23.212927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e50121-0c30-4a0e-894b-accd6d0defba None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:23.213544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e50121-0c30-4a0e-894b-accd6d0defba None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:23.213544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e50121-0c30-4a0e-894b-accd6d0defba None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:23.218771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 693/1404] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:19:23 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:23.220253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3401827c-6440-4062-bd7b-ce4375c2b526 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:23.221163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3401827c-6440-4062-bd7b-ce4375c2b526 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:23.221445 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3401827c-6440-4062-bd7b-ce4375c2b526 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:23.303927 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-3401827c-6440-4062-bd7b-ce4375c2b526 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:23.306032 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-3401827c-6440-4062-bd7b-ce4375c2b526 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:23.408076 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:23.417091 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 712/1405] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:19:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:19:23.729274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e50121-0c30-4a0e-894b-accd6d0defba None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:23.730746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-52e50121-0c30-4a0e-894b-accd6d0defba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:23.896105 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e50121-0c30-4a0e-894b-accd6d0defba None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:23.952605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:24.030758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58492dbe-54fa-4efc-856e-9cfff2213cd4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:24.031409 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58492dbe-54fa-4efc-856e-9cfff2213cd4 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:24.031692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58492dbe-54fa-4efc-856e-9cfff2213cd4 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:24.031970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58492dbe-54fa-4efc-856e-9cfff2213cd4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:24.033677 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-58492dbe-54fa-4efc-856e-9cfff2213cd4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:24.159315 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58492dbe-54fa-4efc-856e-9cfff2213cd4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:24.160975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58492dbe-54fa-4efc-856e-9cfff2213cd4 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:24.183252 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 694/1406] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 455 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:19:26.226571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc4f73b-d99b-4499-a3eb-03bc1edd6b94 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:26.227222 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc4f73b-d99b-4499-a3eb-03bc1edd6b94 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:26.227222 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc4f73b-d99b-4499-a3eb-03bc1edd6b94 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:26.228492 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 713/1407] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:19:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:19:26.233956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9884c503-1225-4533-ab0a-5ada8f648f27 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:26.234413 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9884c503-1225-4533-ab0a-5ada8f648f27 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:26.234519 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9884c503-1225-4533-ab0a-5ada8f648f27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:26.304457 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-9884c503-1225-4533-ab0a-5ada8f648f27 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:26.305778 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9884c503-1225-4533-ab0a-5ada8f648f27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:26.308186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:26.416894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 695/1408] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:19:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:26.710365 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc4f73b-d99b-4499-a3eb-03bc1edd6b94 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:26.712517 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-fbc4f73b-d99b-4499-a3eb-03bc1edd6b94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:26.899412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc4f73b-d99b-4499-a3eb-03bc1edd6b94 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:26.982605 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:27.036603 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de3f942-42d8-4986-9469-be0b12e53441 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:27.037342 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de3f942-42d8-4986-9469-be0b12e53441 service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:27.037641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de3f942-42d8-4986-9469-be0b12e53441 service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:27.038113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de3f942-42d8-4986-9469-be0b12e53441 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:27.040276 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-0de3f942-42d8-4986-9469-be0b12e53441 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:27.149926 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de3f942-42d8-4986-9469-be0b12e53441 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:27.151525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de3f942-42d8-4986-9469-be0b12e53441 service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:27.174573 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 714/1409] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 465 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:19:36.414787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=42 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:37.170508 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=64 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:43.262012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd91df3-de2c-45ef-bcd3-47949d931293 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:43.262589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd91df3-de2c-45ef-bcd3-47949d931293 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:43.262589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd91df3-de2c-45ef-bcd3-47949d931293 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:43.263943 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 696/1410] 10.222.0.85 () {54 vars in 918 bytes} [Tue Aug 30 04:19:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:43.268962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713c5cb4-850b-4bd3-bb3d-81e25962d3f5 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:43.269395 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713c5cb4-850b-4bd3-bb3d-81e25962d3f5 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:43.269555 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713c5cb4-850b-4bd3-bb3d-81e25962d3f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:43.345426 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-713c5cb4-850b-4bd3-bb3d-81e25962d3f5 None None] MFA Rules not processed for user `359d79cbfcad49beba2b495066824128`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:43.346643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-713c5cb4-850b-4bd3-bb3d-81e25962d3f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:43.454067 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 715/1411] 10.222.0.85 () {58 vars in 1015 bytes} [Tue Aug 30 04:19:43 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:43.469177 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e154ff3d-4a77-4790-a343-9a9c2538c0b4 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:43.469458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e154ff3d-4a77-4790-a343-9a9c2538c0b4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:43.469729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e154ff3d-4a77-4790-a343-9a9c2538c0b4 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:43.471088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 697/1412] 10.222.0.85 () {54 vars in 918 bytes} [Tue Aug 30 04:19:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:43.475653 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc325f3-b694-4ab6-82f3-bf305d1c1550 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:43.475946 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc325f3-b694-4ab6-82f3-bf305d1c1550 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:43.476761 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc325f3-b694-4ab6-82f3-bf305d1c1550 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:43.548789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:43.551668 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-fdc325f3-b694-4ab6-82f3-bf305d1c1550 None None] MFA Rules not processed for user `4bc4d455ebf041d488b1199ca58358a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:43.553468 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-fdc325f3-b694-4ab6-82f3-bf305d1c1550 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:43.668775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 716/1413] 10.222.0.85 () {58 vars in 1015 bytes} [Tue Aug 30 04:19:43 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:46.088086 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fc7eba-06d3-4370-80a9-4e18368da9b0 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:46.088664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fc7eba-06d3-4370-80a9-4e18368da9b0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:46.088787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fc7eba-06d3-4370-80a9-4e18368da9b0 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:46.090219 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 698/1414] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 04:19:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:46.095677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104b01e4-2459-4014-a6b5-5e5451be0d75 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:46.096114 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104b01e4-2459-4014-a6b5-5e5451be0d75 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:46.096229 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104b01e4-2459-4014-a6b5-5e5451be0d75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:46.163196 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-104b01e4-2459-4014-a6b5-5e5451be0d75 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:46.164414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-104b01e4-2459-4014-a6b5-5e5451be0d75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:46.219119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:46.263490 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 717/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 04:19:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:19:46.554966 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fc7eba-06d3-4370-80a9-4e18368da9b0 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:46.556712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-89fc7eba-06d3-4370-80a9-4e18368da9b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:46.702473 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fc7eba-06d3-4370-80a9-4e18368da9b0 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:46.775987 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:46.816306 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96f77bc-bc1e-4b98-845e-2b6d1fc39cab service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:46.816833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96f77bc-bc1e-4b98-845e-2b6d1fc39cab service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:46.817141 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96f77bc-bc1e-4b98-845e-2b6d1fc39cab service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:46.817417 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96f77bc-bc1e-4b98-845e-2b6d1fc39cab service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:46.819064 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c96f77bc-bc1e-4b98-845e-2b6d1fc39cab service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:46.910237 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96f77bc-bc1e-4b98-845e-2b6d1fc39cab service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:46.911863 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96f77bc-bc1e-4b98-845e-2b6d1fc39cab service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:46.930957 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 699/1416] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:19:47.848396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c011edf-fa4b-48b3-be24-22151739b5f7 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:47.848955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c011edf-fa4b-48b3-be24-22151739b5f7 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:47.848955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c011edf-fa4b-48b3-be24-22151739b5f7 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:47.850871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 718/1417] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 04:19:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:47.857056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738b39d5-2e9b-4166-b314-17d65ffccdd6 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:47.857523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738b39d5-2e9b-4166-b314-17d65ffccdd6 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:47.857523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738b39d5-2e9b-4166-b314-17d65ffccdd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:47.935577 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-738b39d5-2e9b-4166-b314-17d65ffccdd6 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:47.936842 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-738b39d5-2e9b-4166-b314-17d65ffccdd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:47.964196 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:48.035773 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 700/1418] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 04:19:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:48.186174 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c011edf-fa4b-48b3-be24-22151739b5f7 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:48.187704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-0c011edf-fa4b-48b3-be24-22151739b5f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:48.337766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c011edf-fa4b-48b3-be24-22151739b5f7 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:48.378766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:48.453743 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72439127-01a0-4053-8731-e2e4729c248a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:48.454356 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72439127-01a0-4053-8731-e2e4729c248a service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:48.454636 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72439127-01a0-4053-8731-e2e4729c248a service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:48.454908 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72439127-01a0-4053-8731-e2e4729c248a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:48.456542 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-72439127-01a0-4053-8731-e2e4729c248a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:48.549209 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72439127-01a0-4053-8731-e2e4729c248a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:48.550928 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72439127-01a0-4053-8731-e2e4729c248a service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:48.569145 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 719/1419] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:19:49.460564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79c8cb6-166c-4346-bab1-61d0e3c04f6a None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:49.461161 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79c8cb6-166c-4346-bab1-61d0e3c04f6a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:49.461271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79c8cb6-166c-4346-bab1-61d0e3c04f6a None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:49.462546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 701/1420] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 04:19:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:49.467812 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8c7012-5ed1-4cc7-b0b5-2185aca82a18 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:49.468243 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8c7012-5ed1-4cc7-b0b5-2185aca82a18 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:49.468372 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8c7012-5ed1-4cc7-b0b5-2185aca82a18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:49.524276 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:49.546593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-9b8c7012-5ed1-4cc7-b0b5-2185aca82a18 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:49.547835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-9b8c7012-5ed1-4cc7-b0b5-2185aca82a18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:49.646529 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 720/1421] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 04:19:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:49.799888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79c8cb6-166c-4346-bab1-61d0e3c04f6a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:49.801358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e79c8cb6-166c-4346-bab1-61d0e3c04f6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:49.951658 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79c8cb6-166c-4346-bab1-61d0e3c04f6a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:49.963685 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:50.075615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c086cb3-e7cf-46df-aee4-2176233205e5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:50.076131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c086cb3-e7cf-46df-aee4-2176233205e5 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:50.076408 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c086cb3-e7cf-46df-aee4-2176233205e5 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:50.076679 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c086cb3-e7cf-46df-aee4-2176233205e5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:50.078311 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3c086cb3-e7cf-46df-aee4-2176233205e5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:50.171087 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c086cb3-e7cf-46df-aee4-2176233205e5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:50.172703 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c086cb3-e7cf-46df-aee4-2176233205e5 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:50.191899 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 702/1422] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:19:51.051809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59ff63f-6477-4ea1-bd82-2b5c0792a63e None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:51.052590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59ff63f-6477-4ea1-bd82-2b5c0792a63e None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:51.052699 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59ff63f-6477-4ea1-bd82-2b5c0792a63e None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:51.054034 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 721/1423] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 04:19:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:51.059713 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368ca066-3078-4287-88bb-33c096eaac90 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:51.060129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368ca066-3078-4287-88bb-33c096eaac90 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:51.060292 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368ca066-3078-4287-88bb-33c096eaac90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:51.080073 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:51.148910 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-368ca066-3078-4287-88bb-33c096eaac90 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:51.150205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-368ca066-3078-4287-88bb-33c096eaac90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:51.247337 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 703/1424] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 04:19:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:51.397080 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59ff63f-6477-4ea1-bd82-2b5c0792a63e None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:51.399122 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e59ff63f-6477-4ea1-bd82-2b5c0792a63e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:51.510826 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:51.548078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59ff63f-6477-4ea1-bd82-2b5c0792a63e None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:51.662037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a60ac43-9bb8-4f2d-8b28-a85eff21f064 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:51.662566 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a60ac43-9bb8-4f2d-8b28-a85eff21f064 service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:51.662846 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a60ac43-9bb8-4f2d-8b28-a85eff21f064 service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:51.663140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a60ac43-9bb8-4f2d-8b28-a85eff21f064 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:51.664742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8a60ac43-9bb8-4f2d-8b28-a85eff21f064 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:51.741644 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:51.757942 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a60ac43-9bb8-4f2d-8b28-a85eff21f064 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:51.759575 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a60ac43-9bb8-4f2d-8b28-a85eff21f064 service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:51.781296 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 722/1425] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:19:52.652516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1141e88a-8bb8-46fb-a81a-e36868c03da4 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:52.653140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1141e88a-8bb8-46fb-a81a-e36868c03da4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:52.653140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1141e88a-8bb8-46fb-a81a-e36868c03da4 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:52.654438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 704/1426] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 04:19:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:52.659483 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe35c98-10c2-42c5-a25f-d5fdf684e867 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:52.659908 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe35c98-10c2-42c5-a25f-d5fdf684e867 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:52.660018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe35c98-10c2-42c5-a25f-d5fdf684e867 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:52.735406 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-ebe35c98-10c2-42c5-a25f-d5fdf684e867 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:52.736622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ebe35c98-10c2-42c5-a25f-d5fdf684e867 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:52.833691 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 723/1427] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 04:19:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:52.994177 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1141e88a-8bb8-46fb-a81a-e36868c03da4 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:52.995687 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1141e88a-8bb8-46fb-a81a-e36868c03da4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:53.081027 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:53.145332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1141e88a-8bb8-46fb-a81a-e36868c03da4 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:53.261228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f33c951-ada0-4772-82ed-fc64cafa07c0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:53.261752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f33c951-ada0-4772-82ed-fc64cafa07c0 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:53.262032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f33c951-ada0-4772-82ed-fc64cafa07c0 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:53.262303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f33c951-ada0-4772-82ed-fc64cafa07c0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:53.263912 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1f33c951-ada0-4772-82ed-fc64cafa07c0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:53.311952 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:53.356468 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f33c951-ada0-4772-82ed-fc64cafa07c0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:53.358129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f33c951-ada0-4772-82ed-fc64cafa07c0 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:53.376522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 705/1428] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:19:54.191265 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde79bc1-2240-465d-ae50-6a9fe46256f4 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:54.191828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde79bc1-2240-465d-ae50-6a9fe46256f4 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:54.191828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde79bc1-2240-465d-ae50-6a9fe46256f4 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:54.193308 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 724/1429] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 04:19:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:54.198899 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e3a487-add8-48a2-b4a2-dbdb28b3631e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:54.199355 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e3a487-add8-48a2-b4a2-dbdb28b3631e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:54.199466 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e3a487-add8-48a2-b4a2-dbdb28b3631e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:54.275699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c7e3a487-add8-48a2-b4a2-dbdb28b3631e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:54.276911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c7e3a487-add8-48a2-b4a2-dbdb28b3631e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:54.376798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 706/1430] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 04:19:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:54.526514 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde79bc1-2240-465d-ae50-6a9fe46256f4 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:54.528044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-fde79bc1-2240-465d-ae50-6a9fe46256f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:54.577103 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:54.675469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde79bc1-2240-465d-ae50-6a9fe46256f4 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:54.788030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dfbddb-074c-4e34-a06b-a5489e46107f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:54.788638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dfbddb-074c-4e34-a06b-a5489e46107f service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:54.788920 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dfbddb-074c-4e34-a06b-a5489e46107f service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:54.789224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dfbddb-074c-4e34-a06b-a5489e46107f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:54.790840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-70dfbddb-074c-4e34-a06b-a5489e46107f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:54.797781 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:54.881607 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dfbddb-074c-4e34-a06b-a5489e46107f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:54.883713 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dfbddb-074c-4e34-a06b-a5489e46107f service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:54.903134 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 725/1431] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:19:55.712578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c8139b-de6d-4051-ab76-3f96e559392d None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:55.713173 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c8139b-de6d-4051-ab76-3f96e559392d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:55.713173 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c8139b-de6d-4051-ab76-3f96e559392d None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:55.714497 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 707/1432] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 04:19:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:55.719470 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2014752-126d-40fd-8d79-9b9d6ff97103 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:55.719928 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2014752-126d-40fd-8d79-9b9d6ff97103 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:55.720036 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2014752-126d-40fd-8d79-9b9d6ff97103 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:55.796405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-a2014752-126d-40fd-8d79-9b9d6ff97103 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:55.797635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-a2014752-126d-40fd-8d79-9b9d6ff97103 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:55.866422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:55.896113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 726/1433] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 04:19:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:56.049937 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c8139b-de6d-4051-ab76-3f96e559392d None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:56.051467 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-86c8139b-de6d-4051-ab76-3f96e559392d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:56.068843 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:56.203667 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c8139b-de6d-4051-ab76-3f96e559392d None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:56.290635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:56.316547 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6feb80-9754-4662-8ecb-89030b04f78d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:56.317188 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6feb80-9754-4662-8ecb-89030b04f78d service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:56.317468 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6feb80-9754-4662-8ecb-89030b04f78d service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:56.317737 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6feb80-9754-4662-8ecb-89030b04f78d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:56.319883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fb6feb80-9754-4662-8ecb-89030b04f78d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:56.410217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6feb80-9754-4662-8ecb-89030b04f78d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:56.411828 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6feb80-9754-4662-8ecb-89030b04f78d service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:56.433057 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 708/1434] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:19:57.260675 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e21738-11af-42c9-aa1b-987d8e596db9 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:57.261277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e21738-11af-42c9-aa1b-987d8e596db9 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:57.261277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e21738-11af-42c9-aa1b-987d8e596db9 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:57.262643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 727/1435] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 04:19:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:57.267672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c29d771-38e3-4cad-8560-2ed77226c4a4 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:57.268126 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c29d771-38e3-4cad-8560-2ed77226c4a4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:57.268228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c29d771-38e3-4cad-8560-2ed77226c4a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:57.344505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-2c29d771-38e3-4cad-8560-2ed77226c4a4 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:57.345741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2c29d771-38e3-4cad-8560-2ed77226c4a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:57.388084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:57.443632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 709/1436] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 04:19:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:57.596455 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e21738-11af-42c9-aa1b-987d8e596db9 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:57.598241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-37e21738-11af-42c9-aa1b-987d8e596db9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:57.750021 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e21738-11af-42c9-aa1b-987d8e596db9 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:57.802640 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:57.865250 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df431957-ad32-4dc1-b809-ce0f0e88df83 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:57.866091 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df431957-ad32-4dc1-b809-ce0f0e88df83 service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:57.866830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df431957-ad32-4dc1-b809-ce0f0e88df83 service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:57.867079 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df431957-ad32-4dc1-b809-ce0f0e88df83 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:57.868712 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-df431957-ad32-4dc1-b809-ce0f0e88df83 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:57.957257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df431957-ad32-4dc1-b809-ce0f0e88df83 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:57.958844 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df431957-ad32-4dc1-b809-ce0f0e88df83 service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:57.977406 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 728/1437] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:19:58.791695 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688228e1-ab90-46b0-a7d6-ed383c0522b8 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:58.792268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688228e1-ab90-46b0-a7d6-ed383c0522b8 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:58.792268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688228e1-ab90-46b0-a7d6-ed383c0522b8 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:58.793679 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 710/1438] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 04:19:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:19:58.798484 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b4dddb-8d9c-447e-8b76-978fdca94505 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:58.798881 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b4dddb-8d9c-447e-8b76-978fdca94505 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:58.799037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b4dddb-8d9c-447e-8b76-978fdca94505 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:58.875585 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-f2b4dddb-8d9c-447e-8b76-978fdca94505 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:19:58.876805 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f2b4dddb-8d9c-447e-8b76-978fdca94505 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:58.878882 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:58.978958 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 729/1439] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 04:19:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:19:59.128899 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688228e1-ab90-46b0-a7d6-ed383c0522b8 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:19:59.130492 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-688228e1-ab90-46b0-a7d6-ed383c0522b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:59.278553 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688228e1-ab90-46b0-a7d6-ed383c0522b8 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:19:59.306424 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:19:59.393343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306750bb-643e-46dd-98f5-32281110487c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:19:59.393863 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306750bb-643e-46dd-98f5-32281110487c service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:19:59.394144 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306750bb-643e-46dd-98f5-32281110487c service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:19:59.394413 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306750bb-643e-46dd-98f5-32281110487c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:19:59.396021 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-306750bb-643e-46dd-98f5-32281110487c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:19:59.485283 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306750bb-643e-46dd-98f5-32281110487c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:19:59.486903 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306750bb-643e-46dd-98f5-32281110487c service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:19:59.505541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 711/1440] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:19:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:20:01.336764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd76aa8c-00b0-496d-aa55-fad3a7efbe9c None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:01.337362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd76aa8c-00b0-496d-aa55-fad3a7efbe9c None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:01.337362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd76aa8c-00b0-496d-aa55-fad3a7efbe9c None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:01.338757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 730/1441] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 04:20:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:20:01.344000 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b914f0-9295-4e84-9a24-238e722f67df None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:01.344492 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b914f0-9295-4e84-9a24-238e722f67df None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:01.344612 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b914f0-9295-4e84-9a24-238e722f67df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:01.386168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:01.413236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-26b914f0-9295-4e84-9a24-238e722f67df None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:20:01.414504 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-26b914f0-9295-4e84-9a24-238e722f67df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:01.512522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 712/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 04:20:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:20:01.798985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd76aa8c-00b0-496d-aa55-fad3a7efbe9c None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:20:01.800531 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-fd76aa8c-00b0-496d-aa55-fad3a7efbe9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:01.905589 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd76aa8c-00b0-496d-aa55-fad3a7efbe9c None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:20:01.938130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:01.981537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5edef0-c0a9-4b05-a969-fd1f855b86bb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:20:01.982149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5edef0-c0a9-4b05-a969-fd1f855b86bb service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:01.982426 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5edef0-c0a9-4b05-a969-fd1f855b86bb service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:01.982695 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5edef0-c0a9-4b05-a969-fd1f855b86bb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:01.984310 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-7f5edef0-c0a9-4b05-a969-fd1f855b86bb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:02.091975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5edef0-c0a9-4b05-a969-fd1f855b86bb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:20:02.093376 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5edef0-c0a9-4b05-a969-fd1f855b86bb service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:20:02.112398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 731/1443] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:20:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:20:04.278643 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30660736-2112-425b-afa7-d7adc04b6f95 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:04.279209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30660736-2112-425b-afa7-d7adc04b6f95 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:04.279209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30660736-2112-425b-afa7-d7adc04b6f95 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:04.280550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 713/1444] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 04:20:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:20:04.286062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d23fe8-d336-4ab7-bc05-b0aa8329516b None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:04.286472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d23fe8-d336-4ab7-bc05-b0aa8329516b None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:04.286624 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d23fe8-d336-4ab7-bc05-b0aa8329516b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:04.361477 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-78d23fe8-d336-4ab7-bc05-b0aa8329516b None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:20:04.362602 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-78d23fe8-d336-4ab7-bc05-b0aa8329516b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:04.375465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:04.461800 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 732/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 04:20:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:20:04.745734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30660736-2112-425b-afa7-d7adc04b6f95 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:20:04.747354 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-30660736-2112-425b-afa7-d7adc04b6f95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:04.851938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30660736-2112-425b-afa7-d7adc04b6f95 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:20:04.864166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:04.926896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60620101-98ff-4343-a460-1030dbab8fee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:20:04.927279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60620101-98ff-4343-a460-1030dbab8fee service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:04.927560 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60620101-98ff-4343-a460-1030dbab8fee service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:04.927833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60620101-98ff-4343-a460-1030dbab8fee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:04.930013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-60620101-98ff-4343-a460-1030dbab8fee service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:05.042336 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60620101-98ff-4343-a460-1030dbab8fee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:20:05.043703 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60620101-98ff-4343-a460-1030dbab8fee service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:20:05.061476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 714/1446] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:20:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:20:07.305501 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f7d9f1-7c13-4691-9dcf-b367d68b5706 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:07.306067 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f7d9f1-7c13-4691-9dcf-b367d68b5706 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:07.306067 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f7d9f1-7c13-4691-9dcf-b367d68b5706 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:07.307420 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 733/1447] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 04:20:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:20:07.312787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c733019b-44fd-40d6-b4aa-c9c955ce3f40 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:07.313325 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c733019b-44fd-40d6-b4aa-c9c955ce3f40 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:07.313454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c733019b-44fd-40d6-b4aa-c9c955ce3f40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:07.364746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:07.384982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c733019b-44fd-40d6-b4aa-c9c955ce3f40 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:20:07.386220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c733019b-44fd-40d6-b4aa-c9c955ce3f40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:07.483772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 715/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 04:20:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:20:07.776505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f7d9f1-7c13-4691-9dcf-b367d68b5706 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:20:07.778163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-17f7d9f1-7c13-4691-9dcf-b367d68b5706 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:07.887459 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f7d9f1-7c13-4691-9dcf-b367d68b5706 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:20:07.928018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:07.963621 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321d1ce0-965c-49c7-a168-e8d1e0b90b45 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:20:07.964223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321d1ce0-965c-49c7-a168-e8d1e0b90b45 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:07.964499 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321d1ce0-965c-49c7-a168-e8d1e0b90b45 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:07.964767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321d1ce0-965c-49c7-a168-e8d1e0b90b45 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:07.966513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-321d1ce0-965c-49c7-a168-e8d1e0b90b45 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:08.079665 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321d1ce0-965c-49c7-a168-e8d1e0b90b45 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:20:08.081152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321d1ce0-965c-49c7-a168-e8d1e0b90b45 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:20:08.099568 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 734/1449] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:20:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:20:10.164790 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6532014b-e846-49cb-b817-4f2014576e7b None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:10.165375 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6532014b-e846-49cb-b817-4f2014576e7b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:10.165375 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6532014b-e846-49cb-b817-4f2014576e7b None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:10.166713 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 716/1450] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 04:20:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:20:10.171922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffef5e1-240f-48b7-bf9d-ecc73970ab8c None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:10.172325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffef5e1-240f-48b7-bf9d-ecc73970ab8c None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:10.172475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffef5e1-240f-48b7-bf9d-ecc73970ab8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:10.238490 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-5ffef5e1-240f-48b7-bf9d-ecc73970ab8c None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:20:10.239708 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-5ffef5e1-240f-48b7-bf9d-ecc73970ab8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:10.260946 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:10.339218 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 735/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 04:20:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:20:10.630565 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6532014b-e846-49cb-b817-4f2014576e7b None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:20:10.632253 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6532014b-e846-49cb-b817-4f2014576e7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:10.736550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6532014b-e846-49cb-b817-4f2014576e7b None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:20:10.757280 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:10.811385 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec4354c-088b-4fb6-8d2f-5862a6a3e551 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:20:10.811998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec4354c-088b-4fb6-8d2f-5862a6a3e551 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:10.812282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec4354c-088b-4fb6-8d2f-5862a6a3e551 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:10.812552 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec4354c-088b-4fb6-8d2f-5862a6a3e551 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:10.814196 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3ec4354c-088b-4fb6-8d2f-5862a6a3e551 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:10.924982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec4354c-088b-4fb6-8d2f-5862a6a3e551 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:20:10.926391 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec4354c-088b-4fb6-8d2f-5862a6a3e551 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:20:10.945854 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 717/1452] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:20:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:20:12.948414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da07c23-544e-4fdd-b1b3-f22d425f4b79 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:12.948971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da07c23-544e-4fdd-b1b3-f22d425f4b79 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:12.949094 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da07c23-544e-4fdd-b1b3-f22d425f4b79 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:12.950414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 736/1453] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 04:20:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:20:12.955864 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea0137d-3635-4d68-be66-ab2f6f2ec14e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:12.956268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea0137d-3635-4d68-be66-ab2f6f2ec14e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:12.956406 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea0137d-3635-4d68-be66-ab2f6f2ec14e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:13.027182 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:13.030549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4ea0137d-3635-4d68-be66-ab2f6f2ec14e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:20:13.032515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4ea0137d-3635-4d68-be66-ab2f6f2ec14e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:13.142400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 718/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 04:20:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:20:13.434033 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da07c23-544e-4fdd-b1b3-f22d425f4b79 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:20:13.435583 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-1da07c23-544e-4fdd-b1b3-f22d425f4b79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:13.545703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da07c23-544e-4fdd-b1b3-f22d425f4b79 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:20:13.597728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:13.625676 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c5ee85-8418-42d7-b2f3-9c2b99cf0b6e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:20:13.626291 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c5ee85-8418-42d7-b2f3-9c2b99cf0b6e service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:13.626568 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c5ee85-8418-42d7-b2f3-9c2b99cf0b6e service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:13.626834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c5ee85-8418-42d7-b2f3-9c2b99cf0b6e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:13.628448 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f4c5ee85-8418-42d7-b2f3-9c2b99cf0b6e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:13.739421 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c5ee85-8418-42d7-b2f3-9c2b99cf0b6e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:20:13.741432 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c5ee85-8418-42d7-b2f3-9c2b99cf0b6e service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:20:13.760033 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 737/1455] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:20:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:20:15.735232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5115ee-68bc-4218-ac04-053b0e5395f1 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:15.735799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5115ee-68bc-4218-ac04-053b0e5395f1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:15.735799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5115ee-68bc-4218-ac04-053b0e5395f1 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:15.737177 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 719/1456] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 04:20:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:20:15.743140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2dcb5c-9525-4267-ade4-896c08a4d1e8 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:15.743585 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2dcb5c-9525-4267-ade4-896c08a4d1e8 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:15.743698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2dcb5c-9525-4267-ade4-896c08a4d1e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:15.811640 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-da2dcb5c-9525-4267-ade4-896c08a4d1e8 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:20:15.812854 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-da2dcb5c-9525-4267-ade4-896c08a4d1e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:15.843701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:15.919176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 738/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 04:20:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:20:16.208197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5115ee-68bc-4218-ac04-053b0e5395f1 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:20:16.209781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-df5115ee-68bc-4218-ac04-053b0e5395f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:16.314099 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5115ee-68bc-4218-ac04-053b0e5395f1 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:20:16.342881 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:16.390680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f6ae63-b8eb-4718-81dd-eecb63a44761 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:20:16.391293 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f6ae63-b8eb-4718-81dd-eecb63a44761 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:16.391573 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f6ae63-b8eb-4718-81dd-eecb63a44761 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:16.391843 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f6ae63-b8eb-4718-81dd-eecb63a44761 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:16.393576 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c5f6ae63-b8eb-4718-81dd-eecb63a44761 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:16.504403 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f6ae63-b8eb-4718-81dd-eecb63a44761 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:20:16.505787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f6ae63-b8eb-4718-81dd-eecb63a44761 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:20:16.523455 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 720/1458] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:20:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:20:18.485535 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63958b17-4055-4387-b422-31f0ac6bde61 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:18.486100 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63958b17-4055-4387-b422-31f0ac6bde61 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:18.486100 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63958b17-4055-4387-b422-31f0ac6bde61 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:18.487458 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 739/1459] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 04:20:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:20:18.493414 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b269c78-9b25-4e82-af2f-ac54fa699539 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:18.493854 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b269c78-9b25-4e82-af2f-ac54fa699539 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:18.493986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b269c78-9b25-4e82-af2f-ac54fa699539 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:18.560093 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5b269c78-9b25-4e82-af2f-ac54fa699539 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:20:18.561334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5b269c78-9b25-4e82-af2f-ac54fa699539 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:18.570234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:18.659383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 721/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 04:20:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:20:18.958656 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63958b17-4055-4387-b422-31f0ac6bde61 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:20:18.960187 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-63958b17-4055-4387-b422-31f0ac6bde61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:19.066288 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63958b17-4055-4387-b422-31f0ac6bde61 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:20:19.122100 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:19.141929 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6e553f-4270-4361-85a7-861d22542cf9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:20:19.142543 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6e553f-4270-4361-85a7-861d22542cf9 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:19.142821 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6e553f-4270-4361-85a7-861d22542cf9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:19.143093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6e553f-4270-4361-85a7-861d22542cf9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:19.144765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-3b6e553f-4270-4361-85a7-861d22542cf9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:19.375212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6e553f-4270-4361-85a7-861d22542cf9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:20:19.376551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6e553f-4270-4361-85a7-861d22542cf9 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:20:19.394838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 740/1461] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:20:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 437 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:20:21.357021 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9cd139-b1fe-40d8-ac05-e42d3442e939 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:21.357605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9cd139-b1fe-40d8-ac05-e42d3442e939 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:21.357605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9cd139-b1fe-40d8-ac05-e42d3442e939 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:21.358968 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 722/1462] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 04:20:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:20:21.364233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfae655-9c1b-4837-9e7d-d4e8c9790a80 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:21.364649 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfae655-9c1b-4837-9e7d-d4e8c9790a80 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:21.364747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfae655-9c1b-4837-9e7d-d4e8c9790a80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:21.430000 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-2dfae655-9c1b-4837-9e7d-d4e8c9790a80 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:20:21.431198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-2dfae655-9c1b-4837-9e7d-d4e8c9790a80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:21.469397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:21.527881 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 741/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 04:20:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:20:21.808104 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9cd139-b1fe-40d8-ac05-e42d3442e939 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:20:21.809701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9b9cd139-b1fe-40d8-ac05-e42d3442e939 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:21.915170 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9cd139-b1fe-40d8-ac05-e42d3442e939 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:20:21.952710 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:21.989551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0401fba4-52ee-4959-b93b-cda59984b02b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:20:21.990180 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0401fba4-52ee-4959-b93b-cda59984b02b service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:21.991079 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0401fba4-52ee-4959-b93b-cda59984b02b service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:21.991360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0401fba4-52ee-4959-b93b-cda59984b02b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:21.992953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0401fba4-52ee-4959-b93b-cda59984b02b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:22.101611 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0401fba4-52ee-4959-b93b-cda59984b02b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:20:22.103077 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0401fba4-52ee-4959-b93b-cda59984b02b service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:20:22.127233 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 723/1464] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:20:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:20:24.150460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af545181-ead9-47d9-8613-17d19238cb68 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:24.151019 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af545181-ead9-47d9-8613-17d19238cb68 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:24.151019 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af545181-ead9-47d9-8613-17d19238cb68 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:24.152340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 742/1465] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 04:20:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:20:24.157588 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6c37eb-193f-4871-8cd6-c44297a95203 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:24.158032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6c37eb-193f-4871-8cd6-c44297a95203 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:24.158137 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6c37eb-193f-4871-8cd6-c44297a95203 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:24.226531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5e6c37eb-193f-4871-8cd6-c44297a95203 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:20:24.227753 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5e6c37eb-193f-4871-8cd6-c44297a95203 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:24.246176 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:24.326177 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 724/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 04:20:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:20:24.613529 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af545181-ead9-47d9-8613-17d19238cb68 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:20:24.615063 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-af545181-ead9-47d9-8613-17d19238cb68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:24.718880 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af545181-ead9-47d9-8613-17d19238cb68 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:20:24.785119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:24.792374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acf0aa2-1c56-4020-8311-19c4e2690d7e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:20:24.792974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acf0aa2-1c56-4020-8311-19c4e2690d7e service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:24.793297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acf0aa2-1c56-4020-8311-19c4e2690d7e service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:24.793564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acf0aa2-1c56-4020-8311-19c4e2690d7e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:24.795161 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-9acf0aa2-1c56-4020-8311-19c4e2690d7e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:24.905123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9acf0aa2-1c56-4020-8311-19c4e2690d7e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:20:24.906482 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9acf0aa2-1c56-4020-8311-19c4e2690d7e service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:20:24.924012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 743/1467] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:20:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:20:26.858926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee45d9ca-fa73-4c70-9dee-c4a6596217aa None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:26.859456 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee45d9ca-fa73-4c70-9dee-c4a6596217aa None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:26.859456 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee45d9ca-fa73-4c70-9dee-c4a6596217aa None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:26.860735 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 725/1468] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 04:20:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:20:26.866074 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902717c0-53c6-43a6-8e55-0a3cba0154ec None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:26.866521 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902717c0-53c6-43a6-8e55-0a3cba0154ec None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:26.866632 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902717c0-53c6-43a6-8e55-0a3cba0154ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:26.932045 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-902717c0-53c6-43a6-8e55-0a3cba0154ec None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:20:26.933359 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-902717c0-53c6-43a6-8e55-0a3cba0154ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:26.982046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:27.031854 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 744/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 04:20:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:20:27.320718 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee45d9ca-fa73-4c70-9dee-c4a6596217aa None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:20:27.322337 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ee45d9ca-fa73-4c70-9dee-c4a6596217aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:27.429264 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee45d9ca-fa73-4c70-9dee-c4a6596217aa None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:20:27.477543 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:27.503696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a33794-b394-4c3a-b7e3-448e0fab394f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:20:27.504312 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a33794-b394-4c3a-b7e3-448e0fab394f service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:27.504590 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a33794-b394-4c3a-b7e3-448e0fab394f service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:27.504861 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a33794-b394-4c3a-b7e3-448e0fab394f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:27.506549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-58a33794-b394-4c3a-b7e3-448e0fab394f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:27.620483 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a33794-b394-4c3a-b7e3-448e0fab394f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:20:27.621963 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a33794-b394-4c3a-b7e3-448e0fab394f service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:20:27.641408 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 726/1470] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:20:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:20:37.030053 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=22 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:37.638908 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=66 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:47.342730 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e561e967-cc75-4139-9238-1ea904538549 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:47.343300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e561e967-cc75-4139-9238-1ea904538549 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:47.343300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e561e967-cc75-4139-9238-1ea904538549 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:47.344648 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 745/1471] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:20:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:20:47.350586 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15eabefa-0aae-4f78-8b83-57a4b472d665 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:47.351006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15eabefa-0aae-4f78-8b83-57a4b472d665 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:47.351130 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15eabefa-0aae-4f78-8b83-57a4b472d665 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:47.417837 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-15eabefa-0aae-4f78-8b83-57a4b472d665 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:20:47.418977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-15eabefa-0aae-4f78-8b83-57a4b472d665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:47.517937 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 727/1472] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:20:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:20:47.820152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0811f46b-85fe-438f-ad0e-1fbf41ec64a8 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:47.820647 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0811f46b-85fe-438f-ad0e-1fbf41ec64a8 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:47.821003 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0811f46b-85fe-438f-ad0e-1fbf41ec64a8 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:47.822376 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 746/1473] 10.222.0.85 () {54 vars in 967 bytes} [Tue Aug 30 04:20:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:20:47.827156 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a58fb6-719a-4485-80a0-2aeb724d9b05 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:47.827442 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a58fb6-719a-4485-80a0-2aeb724d9b05 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:47.827714 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a58fb6-719a-4485-80a0-2aeb724d9b05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:47.904314 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-95a58fb6-719a-4485-80a0-2aeb724d9b05 None None] MFA Rules not processed for user `354b8228661c4369b3bca2e631f3f140`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:20:47.905570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-95a58fb6-719a-4485-80a0-2aeb724d9b05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:47.926714 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:47.981924 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 728/1474] 10.222.0.85 () {58 vars in 1064 bytes} [Tue Aug 30 04:20:47 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:20:47.989511 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0811f46b-85fe-438f-ad0e-1fbf41ec64a8 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:20:47.991664 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-0811f46b-85fe-438f-ad0e-1fbf41ec64a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:48.099205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0811f46b-85fe-438f-ad0e-1fbf41ec64a8 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:20:48.172087 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b135bd-f970-46a8-b9bc-48cd3d9317fb service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '354b8228661c4369b3bca2e631f3f140', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:20:48.172688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b135bd-f970-46a8-b9bc-48cd3d9317fb service glance] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:48.172966 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b135bd-f970-46a8-b9bc-48cd3d9317fb service glance] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:48.173265 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b135bd-f970-46a8-b9bc-48cd3d9317fb service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:48.174875 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-a1b135bd-f970-46a8-b9bc-48cd3d9317fb service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:48.231517 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:48.289520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b135bd-f970-46a8-b9bc-48cd3d9317fb service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:20:48.290969 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b135bd-f970-46a8-b9bc-48cd3d9317fb service glance] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:20:48.310261 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 747/1475] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:20:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:20:50.337473 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271f74c8-7e95-451a-a843-063e11a398f1 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:50.338035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271f74c8-7e95-451a-a843-063e11a398f1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:50.338035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271f74c8-7e95-451a-a843-063e11a398f1 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:50.339388 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 729/1476] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:20:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:20:50.344880 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ec1aed-bb21-46d8-8b04-dd63652f594d None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:50.345310 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ec1aed-bb21-46d8-8b04-dd63652f594d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:50.345506 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ec1aed-bb21-46d8-8b04-dd63652f594d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:50.414002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-99ec1aed-bb21-46d8-8b04-dd63652f594d None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:20:50.415119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-99ec1aed-bb21-46d8-8b04-dd63652f594d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:50.513477 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 748/1477] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:20:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:20:50.786434 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271f74c8-7e95-451a-a843-063e11a398f1 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:20:50.788038 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-271f74c8-7e95-451a-a843-063e11a398f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:50.915366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271f74c8-7e95-451a-a843-063e11a398f1 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:20:50.964022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:51.009730 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243c20b4-3a7b-4a2f-b06c-cf8931016f0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:20:51.010343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243c20b4-3a7b-4a2f-b06c-cf8931016f0b admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:51.010622 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243c20b4-3a7b-4a2f-b06c-cf8931016f0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:51.010894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243c20b4-3a7b-4a2f-b06c-cf8931016f0b admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:51.011535 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243c20b4-3a7b-4a2f-b06c-cf8931016f0b admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:20:51.013380 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: WARNING py.warnings [None req-243c20b4-3a7b-4a2f-b06c-cf8931016f0b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 04:20:51.013380 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: warnings.warn(msg) Aug 30 04:20:51.013380 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: Aug 30 04:20:51.014552 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243c20b4-3a7b-4a2f-b06c-cf8931016f0b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:20:51.033678 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 730/1478] 10.222.0.85 () {56 vars in 1108 bytes} [Tue Aug 30 04:20:50 2022] GET /identity/v3/projects => generated 2553 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:20:53.020165 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436b3035-db08-4f86-a346-a1df95613aec None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:53.020767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436b3035-db08-4f86-a346-a1df95613aec None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:53.020767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436b3035-db08-4f86-a346-a1df95613aec None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:53.022713 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 749/1479] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:20:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:20:53.028273 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b27e627-9a11-4efc-8d8c-59d43ed5e509 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:53.028708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b27e627-9a11-4efc-8d8c-59d43ed5e509 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:53.028895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b27e627-9a11-4efc-8d8c-59d43ed5e509 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:53.095471 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-7b27e627-9a11-4efc-8d8c-59d43ed5e509 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:20:53.096716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7b27e627-9a11-4efc-8d8c-59d43ed5e509 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:53.195299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 731/1480] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:20:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:20:53.488638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436b3035-db08-4f86-a346-a1df95613aec None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:20:53.490763 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-436b3035-db08-4f86-a346-a1df95613aec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:53.500865 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:53.641168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436b3035-db08-4f86-a346-a1df95613aec None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:20:53.722370 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:53.757229 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac32e445-42a5-45fd-933a-1b5367014c59 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:20:53.757765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac32e445-42a5-45fd-933a-1b5367014c59 service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:53.758046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac32e445-42a5-45fd-933a-1b5367014c59 service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:53.758318 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac32e445-42a5-45fd-933a-1b5367014c59 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:53.759942 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ac32e445-42a5-45fd-933a-1b5367014c59 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:53.848198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac32e445-42a5-45fd-933a-1b5367014c59 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:20:53.850423 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac32e445-42a5-45fd-933a-1b5367014c59 service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:20:53.868152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 750/1481] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:20:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:20:56.003163 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d375c8a6-4d5b-466a-ab2d-47704ba0b3af None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:56.003801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d375c8a6-4d5b-466a-ab2d-47704ba0b3af None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:56.003801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d375c8a6-4d5b-466a-ab2d-47704ba0b3af None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:56.005100 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 732/1482] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:20:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:20:56.011006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56e2b85-65d3-4835-8870-2da067ce75a5 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:56.011424 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56e2b85-65d3-4835-8870-2da067ce75a5 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:56.011551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56e2b85-65d3-4835-8870-2da067ce75a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:56.079417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-c56e2b85-65d3-4835-8870-2da067ce75a5 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:20:56.080642 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c56e2b85-65d3-4835-8870-2da067ce75a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:56.128815 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:56.180238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 751/1483] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:20:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:20:56.469281 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d375c8a6-4d5b-466a-ab2d-47704ba0b3af None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:20:56.470887 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d375c8a6-4d5b-466a-ab2d-47704ba0b3af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:56.497534 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:56.578571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d375c8a6-4d5b-466a-ab2d-47704ba0b3af None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:20:56.651704 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2aebab2-2c7a-494c-8ec8-3085721db393 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:20:56.652305 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2aebab2-2c7a-494c-8ec8-3085721db393 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:56.652582 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2aebab2-2c7a-494c-8ec8-3085721db393 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:56.652851 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2aebab2-2c7a-494c-8ec8-3085721db393 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:56.654490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b2aebab2-2c7a-494c-8ec8-3085721db393 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:56.850901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:56.887811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2aebab2-2c7a-494c-8ec8-3085721db393 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:20:56.889290 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2aebab2-2c7a-494c-8ec8-3085721db393 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:20:56.907097 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 733/1484] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:20:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 438 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:20:58.890053 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f366287-8aa1-486f-a6f9-50abceeacc92 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:58.890677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f366287-8aa1-486f-a6f9-50abceeacc92 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:58.890677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f366287-8aa1-486f-a6f9-50abceeacc92 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:58.891924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 752/1485] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:20:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:20:58.897314 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44429501-20f6-4963-9e1a-1ed49b0a6984 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:58.897759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44429501-20f6-4963-9e1a-1ed49b0a6984 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:58.897865 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44429501-20f6-4963-9e1a-1ed49b0a6984 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:58.964410 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-44429501-20f6-4963-9e1a-1ed49b0a6984 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:20:58.965636 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-44429501-20f6-4963-9e1a-1ed49b0a6984 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:59.061891 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 734/1486] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:20:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:20:59.349384 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f366287-8aa1-486f-a6f9-50abceeacc92 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:20:59.351051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-9f366287-8aa1-486f-a6f9-50abceeacc92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:59.457849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f366287-8aa1-486f-a6f9-50abceeacc92 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:20:59.531773 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:20:59.533513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c01879b-54c0-4d86-843c-fbc2f15ca98d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:20:59.534338 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c01879b-54c0-4d86-843c-fbc2f15ca98d service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:20:59.534656 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c01879b-54c0-4d86-843c-fbc2f15ca98d service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:20:59.535065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c01879b-54c0-4d86-843c-fbc2f15ca98d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:20:59.537765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8c01879b-54c0-4d86-843c-fbc2f15ca98d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:20:59.645817 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c01879b-54c0-4d86-843c-fbc2f15ca98d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:20:59.647858 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c01879b-54c0-4d86-843c-fbc2f15ca98d service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:20:59.666373 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 753/1487] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:20:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:21:01.684883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a335e8ff-e2e8-4c4c-a6d6-1333e3d79c29 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:21:01.685511 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a335e8ff-e2e8-4c4c-a6d6-1333e3d79c29 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:21:01.685511 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a335e8ff-e2e8-4c4c-a6d6-1333e3d79c29 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:21:01.686845 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 735/1488] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:21:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:21:01.693967 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0309d3d-a585-47ba-8470-f33192ec88e9 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:21:01.694530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0309d3d-a585-47ba-8470-f33192ec88e9 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:21:01.694636 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0309d3d-a585-47ba-8470-f33192ec88e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:21:01.779784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-f0309d3d-a585-47ba-8470-f33192ec88e9 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:21:01.781018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f0309d3d-a585-47ba-8470-f33192ec88e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:21:01.839823 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:21:01.886451 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 754/1489] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:21:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:21:02.190112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a335e8ff-e2e8-4c4c-a6d6-1333e3d79c29 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:21:02.191641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a335e8ff-e2e8-4c4c-a6d6-1333e3d79c29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:21:02.228771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:21:02.299130 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a335e8ff-e2e8-4c4c-a6d6-1333e3d79c29 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:21:02.374694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c9a685-bfdd-4c23-8a65-507147e20c60 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:21:02.375219 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c9a685-bfdd-4c23-8a65-507147e20c60 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:21:02.375498 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c9a685-bfdd-4c23-8a65-507147e20c60 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:21:02.375782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c9a685-bfdd-4c23-8a65-507147e20c60 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:21:02.377425 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-90c9a685-bfdd-4c23-8a65-507147e20c60 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:21:02.458559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:21:02.489355 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c9a685-bfdd-4c23-8a65-507147e20c60 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:21:02.491493 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c9a685-bfdd-4c23-8a65-507147e20c60 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:21:02.510245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 736/1490] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:21:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:21:04.694450 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d5b480-5e8a-4ab6-8b2f-d6f8926d27ef None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:21:04.695044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d5b480-5e8a-4ab6-8b2f-d6f8926d27ef None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:21:04.695044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d5b480-5e8a-4ab6-8b2f-d6f8926d27ef None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:21:04.696346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 755/1491] 10.222.0.85 () {54 vars in 927 bytes} [Tue Aug 30 04:21:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:21:04.701677 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13827fb-cb07-46a8-9130-f1c6d9cd72d0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:21:04.702142 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13827fb-cb07-46a8-9130-f1c6d9cd72d0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:21:04.702278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13827fb-cb07-46a8-9130-f1c6d9cd72d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:21:04.773779 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-f13827fb-cb07-46a8-9130-f1c6d9cd72d0 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:21:04.775047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f13827fb-cb07-46a8-9130-f1c6d9cd72d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:21:04.874970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 737/1492] 10.222.0.85 () {58 vars in 1024 bytes} [Tue Aug 30 04:21:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:21:05.166697 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d5b480-5e8a-4ab6-8b2f-d6f8926d27ef None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:21:05.168256 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-67d5b480-5e8a-4ab6-8b2f-d6f8926d27ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:21:05.280175 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d5b480-5e8a-4ab6-8b2f-d6f8926d27ef None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:21:05.355752 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19d33ee-e758-4ed4-8709-ac084863d184 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:21:05.356358 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19d33ee-e758-4ed4-8709-ac084863d184 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:21:05.356697 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19d33ee-e758-4ed4-8709-ac084863d184 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:21:05.357425 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19d33ee-e758-4ed4-8709-ac084863d184 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:21:05.358654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b19d33ee-e758-4ed4-8709-ac084863d184 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:21:05.368763 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:21:05.469809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19d33ee-e758-4ed4-8709-ac084863d184 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:21:05.471269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19d33ee-e758-4ed4-8709-ac084863d184 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:21:05.489603 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 756/1493] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:21:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:21:14.872829 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=82 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:21:15.487334 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=50 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:22:22.301489 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247e548b-3394-4bd3-be59-c023d9884131 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:22:22.302089 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247e548b-3394-4bd3-be59-c023d9884131 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:22:22.302089 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247e548b-3394-4bd3-be59-c023d9884131 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:22:22.377396 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-247e548b-3394-4bd3-be59-c023d9884131 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:22:22.379037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-247e548b-3394-4bd3-be59-c023d9884131 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:22:22.474694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 738/1494] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:22:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:22:22.482920 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7533556-1a8f-4929-86e7-930656a7ade9 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:22:22.483419 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7533556-1a8f-4929-86e7-930656a7ade9 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:22:22.483419 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7533556-1a8f-4929-86e7-930656a7ade9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:22:22.558879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-e7533556-1a8f-4929-86e7-930656a7ade9 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:22:22.560102 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e7533556-1a8f-4929-86e7-930656a7ade9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:22:22.657383 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 757/1495] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:22:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:22:22.661950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247e548b-3394-4bd3-be59-c023d9884131 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:22:22.663174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-247e548b-3394-4bd3-be59-c023d9884131 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:22:22.674567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247e548b-3394-4bd3-be59-c023d9884131 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:22:22.735737 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:22:22.771372 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187ab35b-d161-4c6f-b0ec-17bdf0b2cff0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:22:22.771979 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187ab35b-d161-4c6f-b0ec-17bdf0b2cff0 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:22:22.772258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187ab35b-d161-4c6f-b0ec-17bdf0b2cff0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:22:22.772534 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187ab35b-d161-4c6f-b0ec-17bdf0b2cff0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:22:22.773207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187ab35b-d161-4c6f-b0ec-17bdf0b2cff0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:22:22.773509 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187ab35b-d161-4c6f-b0ec-17bdf0b2cff0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:22:22.774523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: WARNING py.warnings [None req-187ab35b-d161-4c6f-b0ec-17bdf0b2cff0 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 30 04:22:22.774523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: warnings.warn(msg) Aug 30 04:22:22.774523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: Aug 30 04:22:22.775399 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187ab35b-d161-4c6f-b0ec-17bdf0b2cff0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:22:22.786416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 739/1496] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:22:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:22:22.794901 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7533556-1a8f-4929-86e7-930656a7ade9 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:22:22.804570 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7533556-1a8f-4929-86e7-930656a7ade9 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:22:22.866181 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:22:22.902836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d5d755-e034-4470-8544-82047222cf53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:22:22.903428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d5d755-e034-4470-8544-82047222cf53 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:22:22.903702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d5d755-e034-4470-8544-82047222cf53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:22:22.903994 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d5d755-e034-4470-8544-82047222cf53 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:22:22.904618 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d5d755-e034-4470-8544-82047222cf53 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:22:22.906368 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: WARNING py.warnings [None req-d9d5d755-e034-4470-8544-82047222cf53 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 04:22:22.906368 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: warnings.warn(msg) Aug 30 04:22:22.906368 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: Aug 30 04:22:22.907615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d5d755-e034-4470-8544-82047222cf53 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:22:22.966341 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 758/1497] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:22:22 2022] POST /identity/v3/projects => generated 379 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:22:22.970705 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187ab35b-d161-4c6f-b0ec-17bdf0b2cff0 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:22:22.972424 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-187ab35b-d161-4c6f-b0ec-17bdf0b2cff0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:22:23.007387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187ab35b-d161-4c6f-b0ec-17bdf0b2cff0 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:22:23.102607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d058f762-423e-45f2-a668-5a320f813bfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:22:23.103216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d058f762-423e-45f2-a668-5a320f813bfc admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:22:23.103498 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d058f762-423e-45f2-a668-5a320f813bfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:22:23.103764 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d058f762-423e-45f2-a668-5a320f813bfc admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:22:23.104388 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d058f762-423e-45f2-a668-5a320f813bfc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:22:23.105450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: WARNING py.warnings [None req-d058f762-423e-45f2-a668-5a320f813bfc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 04:22:23.105450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: warnings.warn(msg) Aug 30 04:22:23.105450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: Aug 30 04:22:23.106666 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d058f762-423e-45f2-a668-5a320f813bfc admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:22:23.150860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 740/1498] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:22:22 2022] POST /identity/v3/users => generated 290 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:22:23.154971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d5d755-e034-4470-8544-82047222cf53 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:22:23.156576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d9d5d755-e034-4470-8544-82047222cf53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:22:23.219051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d5d755-e034-4470-8544-82047222cf53 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:22:23.302342 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:22:23.313503 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04981e85-9bbb-448c-9fac-a8f0e618b43c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:22:23.314112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04981e85-9bbb-448c-9fac-a8f0e618b43c admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:22:23.314390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04981e85-9bbb-448c-9fac-a8f0e618b43c admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:22:23.314658 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04981e85-9bbb-448c-9fac-a8f0e618b43c admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:22:23.315298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04981e85-9bbb-448c-9fac-a8f0e618b43c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:22:23.316818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: WARNING py.warnings [None req-04981e85-9bbb-448c-9fac-a8f0e618b43c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 30 04:22:23.316818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: warnings.warn(msg) Aug 30 04:22:23.316818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: Aug 30 04:22:23.317744 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04981e85-9bbb-448c-9fac-a8f0e618b43c admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:22:23.330186 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 759/1499] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:22:23 2022] GET /identity/v3/roles => generated 1420 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:22:23.334753 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d058f762-423e-45f2-a668-5a320f813bfc admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:22:23.336281 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d058f762-423e-45f2-a668-5a320f813bfc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:22:23.347558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d058f762-423e-45f2-a668-5a320f813bfc admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:22:23.371924 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:22:23.442364 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06da2da-8952-42e3-96a1-7a540b546307 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:22:23.442976 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06da2da-8952-42e3-96a1-7a540b546307 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:22:23.443254 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06da2da-8952-42e3-96a1-7a540b546307 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:22:23.443523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06da2da-8952-42e3-96a1-7a540b546307 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:22:23.444167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06da2da-8952-42e3-96a1-7a540b546307 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:22:23.445222 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: WARNING py.warnings [None req-a06da2da-8952-42e3-96a1-7a540b546307 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 30 04:22:23.445222 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: warnings.warn(msg) Aug 30 04:22:23.445222 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: Aug 30 04:22:23.446111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06da2da-8952-42e3-96a1-7a540b546307 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:22:23.459061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 741/1500] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:22:23 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:22:23.463303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04981e85-9bbb-448c-9fac-a8f0e618b43c admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:22:23.464833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-04981e85-9bbb-448c-9fac-a8f0e618b43c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:22:23.499509 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04981e85-9bbb-448c-9fac-a8f0e618b43c admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:22:23.592245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97bf14a-94aa-44ca-82d5-55ec12fd9090 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:22:23.592903 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97bf14a-94aa-44ca-82d5-55ec12fd9090 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:22:23.593215 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97bf14a-94aa-44ca-82d5-55ec12fd9090 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:22:23.593491 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97bf14a-94aa-44ca-82d5-55ec12fd9090 admin admin] PATH_INFO: `/v3/projects/afacda908741443399130dadaf51076d/users/0a813db89ec64fbc9cd962a96f744ca2/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:22:23.615609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97bf14a-94aa-44ca-82d5-55ec12fd9090 admin admin] RBAC: Authorizing `identity:create_grant(project_id=afacda908741443399130dadaf51076d, user_id=0a813db89ec64fbc9cd962a96f744ca2, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:22:23.616657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: WARNING py.warnings [None req-f97bf14a-94aa-44ca-82d5-55ec12fd9090 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 04:22:23.616657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: warnings.warn(msg) Aug 30 04:22:23.616657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: Aug 30 04:22:23.618630 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97bf14a-94aa-44ca-82d5-55ec12fd9090 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:22:23.643610 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 760/1501] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:22:23 2022] PUT /identity/v3/projects/afacda908741443399130dadaf51076d/users/0a813db89ec64fbc9cd962a96f744ca2/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:22:23.656726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fefb20e-35ab-4959-af4e-3bb2c27d84d1 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:22:23.657201 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fefb20e-35ab-4959-af4e-3bb2c27d84d1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:22:23.657490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fefb20e-35ab-4959-af4e-3bb2c27d84d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:22:23.715967 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-1fefb20e-35ab-4959-af4e-3bb2c27d84d1 None None] MFA Rules not processed for user `0a813db89ec64fbc9cd962a96f744ca2`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:22:23.717228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1fefb20e-35ab-4959-af4e-3bb2c27d84d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:22:23.795431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 742/1502] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:22:23 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:22:23.809636 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870fb967-5be9-4a59-a8df-e732a32d7b0a None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:22:23.809925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870fb967-5be9-4a59-a8df-e732a32d7b0a None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:22:23.810205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870fb967-5be9-4a59-a8df-e732a32d7b0a None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:22:23.811564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 761/1503] 10.222.0.85 () {54 vars in 964 bytes} [Tue Aug 30 04:22:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:22:23.817541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ad2f7a-a577-48c5-904b-ac02c916fc2f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:22:23.817826 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ad2f7a-a577-48c5-904b-ac02c916fc2f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:22:23.818100 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ad2f7a-a577-48c5-904b-ac02c916fc2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:22:23.848404 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:22:23.893661 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-23ad2f7a-a577-48c5-904b-ac02c916fc2f None None] MFA Rules not processed for user `7ef280ce39b14085be329f60b22530d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:22:23.894897 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-23ad2f7a-a577-48c5-904b-ac02c916fc2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:22:23.970103 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 743/1504] 10.222.0.85 () {58 vars in 1061 bytes} [Tue Aug 30 04:22:23 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:22:23.978731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870fb967-5be9-4a59-a8df-e732a32d7b0a None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:22:23.980253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-870fb967-5be9-4a59-a8df-e732a32d7b0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:22:24.031508 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:22:24.087817 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870fb967-5be9-4a59-a8df-e732a32d7b0a None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:22:24.162407 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d95655a-ab31-4c4f-b58f-13b0954b3271 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:22:24.163010 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d95655a-ab31-4c4f-b58f-13b0954b3271 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:22:24.163288 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d95655a-ab31-4c4f-b58f-13b0954b3271 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:22:24.163557 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d95655a-ab31-4c4f-b58f-13b0954b3271 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:22:24.165212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4d95655a-ab31-4c4f-b58f-13b0954b3271 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:22:24.264053 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d95655a-ab31-4c4f-b58f-13b0954b3271 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:22:24.265576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d95655a-ab31-4c4f-b58f-13b0954b3271 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:22:24.269027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:22:24.283629 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 762/1505] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:22:23 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:22:24.329958 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65225e8-9e22-43d5-96c5-50ff98ae222e None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:22:24.330271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65225e8-9e22-43d5-96c5-50ff98ae222e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:22:24.330515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65225e8-9e22-43d5-96c5-50ff98ae222e None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:22:24.331779 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 744/1506] 10.222.0.85 () {52 vars in 830 bytes} [Tue Aug 30 04:22:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:22:24.335451 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d95655a-ab31-4c4f-b58f-13b0954b3271 service cinder] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:22:24.336656 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4d95655a-ab31-4c4f-b58f-13b0954b3271 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:22:24.440043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d95655a-ab31-4c4f-b58f-13b0954b3271 service cinder] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:22:24.536395 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f7e5ee-64e9-40d9-88cb-41957e261588 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:22:24.537121 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f7e5ee-64e9-40d9-88cb-41957e261588 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:22:24.537318 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f7e5ee-64e9-40d9-88cb-41957e261588 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:22:24.537588 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f7e5ee-64e9-40d9-88cb-41957e261588 admin admin] PATH_INFO: `/v3/users/0a813db89ec64fbc9cd962a96f744ca2` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:22:24.550055 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f7e5ee-64e9-40d9-88cb-41957e261588 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0a813db89ec64fbc9cd962a96f744ca2)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:22:24.551083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: WARNING py.warnings [None req-18f7e5ee-64e9-40d9-88cb-41957e261588 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 04:22:24.551083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: warnings.warn(msg) Aug 30 04:22:24.551083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: Aug 30 04:22:24.552345 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f7e5ee-64e9-40d9-88cb-41957e261588 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:22:24.561967 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:22:24.619107 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-18f7e5ee-64e9-40d9-88cb-41957e261588 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0a813db89ec64fbc9cd962a96f744ca2'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:22:24.629592 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-18f7e5ee-64e9-40d9-88cb-41957e261588 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0a813db89ec64fbc9cd962a96f744ca2'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:22:24.630122 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-18f7e5ee-64e9-40d9-88cb-41957e261588 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0a813db89ec64fbc9cd962a96f744ca2'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:22:24.655356 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-18f7e5ee-64e9-40d9-88cb-41957e261588 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0a813db89ec64fbc9cd962a96f744ca2'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:22:24.655826 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-18f7e5ee-64e9-40d9-88cb-41957e261588 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0a813db89ec64fbc9cd962a96f744ca2'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:22:24.676920 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 763/1507] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:22:24 2022] DELETE /identity/v3/users/0a813db89ec64fbc9cd962a96f744ca2 => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:22:24.681346 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf049e26-d79b-41ac-a2e3-afb5c1d7d02a None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:22:24.681826 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf049e26-d79b-41ac-a2e3-afb5c1d7d02a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:22:24.681927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf049e26-d79b-41ac-a2e3-afb5c1d7d02a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:22:24.739138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-bf049e26-d79b-41ac-a2e3-afb5c1d7d02a None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:22:24.740384 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bf049e26-d79b-41ac-a2e3-afb5c1d7d02a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:22:24.817113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:22:24.831215 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 745/1508] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:22:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:22:24.841459 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f7e5ee-64e9-40d9-88cb-41957e261588 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:22:24.842677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-18f7e5ee-64e9-40d9-88cb-41957e261588 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:22:24.943829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f7e5ee-64e9-40d9-88cb-41957e261588 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:22:25.003360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:22:25.024485 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5b0968-62ab-47b1-af06-63240c3bff92 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:22:25.025131 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5b0968-62ab-47b1-af06-63240c3bff92 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:22:25.025415 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5b0968-62ab-47b1-af06-63240c3bff92 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:22:25.025687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5b0968-62ab-47b1-af06-63240c3bff92 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:22:25.027320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-cd5b0968-62ab-47b1-af06-63240c3bff92 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:22:25.137116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5b0968-62ab-47b1-af06-63240c3bff92 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:22:25.138567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5b0968-62ab-47b1-af06-63240c3bff92 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:22:25.156799 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 764/1509] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:22:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:22:25.507369 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf049e26-d79b-41ac-a2e3-afb5c1d7d02a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:22:25.508667 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bf049e26-d79b-41ac-a2e3-afb5c1d7d02a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:22:25.521196 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf049e26-d79b-41ac-a2e3-afb5c1d7d02a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:22:25.621441 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e35f4bb-522c-4aa1-ae03-43f807e17878 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:22:25.622072 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e35f4bb-522c-4aa1-ae03-43f807e17878 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:22:25.622359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e35f4bb-522c-4aa1-ae03-43f807e17878 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:22:25.622633 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e35f4bb-522c-4aa1-ae03-43f807e17878 admin admin] PATH_INFO: `/v3/projects/afacda908741443399130dadaf51076d` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:22:25.634280 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e35f4bb-522c-4aa1-ae03-43f807e17878 admin admin] RBAC: Authorizing `identity:delete_project(project_id=afacda908741443399130dadaf51076d)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:22:25.635430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: WARNING py.warnings [None req-1e35f4bb-522c-4aa1-ae03-43f807e17878 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 04:22:25.635430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: warnings.warn(msg) Aug 30 04:22:25.635430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: Aug 30 04:22:25.636649 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e35f4bb-522c-4aa1-ae03-43f807e17878 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:22:25.697677 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1e35f4bb-522c-4aa1-ae03-43f807e17878 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'afacda908741443399130dadaf51076d', 'request_id': 'req-1e35f4bb-522c-4aa1-ae03-43f807e17878'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:22:25.703840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1e35f4bb-522c-4aa1-ae03-43f807e17878 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'afacda908741443399130dadaf51076d', 'request_id': 'req-1e35f4bb-522c-4aa1-ae03-43f807e17878'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:22:25.714446 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1e35f4bb-522c-4aa1-ae03-43f807e17878 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'afacda908741443399130dadaf51076d', 'request_id': 'req-1e35f4bb-522c-4aa1-ae03-43f807e17878'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:22:25.746888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1e35f4bb-522c-4aa1-ae03-43f807e17878 admin admin] The token cache is being invalidate because project afacda908741443399130dadaf51076d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:22:25.747176 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1e35f4bb-522c-4aa1-ae03-43f807e17878 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:22:25.747608 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1e35f4bb-522c-4aa1-ae03-43f807e17878 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:22:25.749307 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 746/1510] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:22:25 2022] DELETE /identity/v3/projects/afacda908741443399130dadaf51076d => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:22:35.154677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=62 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:22:35.744846 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:24:01.941203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba11dcfb-fa43-4c25-a399-c6bcfc65c6bc None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:01.941811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba11dcfb-fa43-4c25-a399-c6bcfc65c6bc None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:01.941811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba11dcfb-fa43-4c25-a399-c6bcfc65c6bc None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:01.943123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 765/1511] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 04:24:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:24:01.948510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b8a1a3-88fc-46ab-90b7-0242e9c40ecd None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:01.948971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b8a1a3-88fc-46ab-90b7-0242e9c40ecd None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:01.949120 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b8a1a3-88fc-46ab-90b7-0242e9c40ecd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:02.030388 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-32b8a1a3-88fc-46ab-90b7-0242e9c40ecd None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:24:02.031534 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-32b8a1a3-88fc-46ab-90b7-0242e9c40ecd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:24:02.135648 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 747/1512] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 04:24:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:24:02.289859 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba11dcfb-fa43-4c25-a399-c6bcfc65c6bc None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:24:02.291957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ba11dcfb-fa43-4c25-a399-c6bcfc65c6bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:24:02.411590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba11dcfb-fa43-4c25-a399-c6bcfc65c6bc None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:24:02.417606 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fba31d6-6d0c-40e5-928d-f632bf0d435d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:24:02.418230 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fba31d6-6d0c-40e5-928d-f632bf0d435d service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:02.418494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fba31d6-6d0c-40e5-928d-f632bf0d435d service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:02.418763 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fba31d6-6d0c-40e5-928d-f632bf0d435d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:02.426613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fba31d6-6d0c-40e5-928d-f632bf0d435d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:24:02.428192 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fba31d6-6d0c-40e5-928d-f632bf0d435d service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:24:02.437434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 766/1513] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:24:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:24:03.282229 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eeececf-f299-46de-85cf-9dee2b134bd5 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:03.282819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eeececf-f299-46de-85cf-9dee2b134bd5 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:03.282819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eeececf-f299-46de-85cf-9dee2b134bd5 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:03.284138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 748/1514] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 04:24:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:24:03.289194 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7668475-45b4-42ee-a062-b9886878af65 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:03.289654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7668475-45b4-42ee-a062-b9886878af65 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:03.289762 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7668475-45b4-42ee-a062-b9886878af65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:03.309745 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-d7668475-45b4-42ee-a062-b9886878af65 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:24:03.310960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d7668475-45b4-42ee-a062-b9886878af65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:24:03.319932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 767/1515] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 04:24:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:24:03.474843 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eeececf-f299-46de-85cf-9dee2b134bd5 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:24:03.500484 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eeececf-f299-46de-85cf-9dee2b134bd5 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:24:03.510007 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31af26a1-4d97-41d2-bff0-2002b3015ac2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:24:03.510981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31af26a1-4d97-41d2-bff0-2002b3015ac2 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:03.511446 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31af26a1-4d97-41d2-bff0-2002b3015ac2 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:03.511924 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31af26a1-4d97-41d2-bff0-2002b3015ac2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:03.524522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31af26a1-4d97-41d2-bff0-2002b3015ac2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:24:03.526205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31af26a1-4d97-41d2-bff0-2002b3015ac2 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:24:03.536557 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 749/1516] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:24:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:24:06.418375 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814d1cec-993b-4919-be96-963bf08f2070 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:06.418902 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814d1cec-993b-4919-be96-963bf08f2070 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:06.418902 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814d1cec-993b-4919-be96-963bf08f2070 None None] PATH_INFO: `/v3` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:06.420127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 768/1517] 10.222.0.71 () {54 vars in 925 bytes} [Tue Aug 30 04:24:06 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:24:06.429985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de0d76c-0b82-4c94-aa66-bc08d783d330 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:06.430429 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de0d76c-0b82-4c94-aa66-bc08d783d330 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:06.430537 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de0d76c-0b82-4c94-aa66-bc08d783d330 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:06.490133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-7de0d76c-0b82-4c94-aa66-bc08d783d330 None None] MFA Rules not processed for user `359d79cbfcad49beba2b495066824128`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:24:06.492008 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7de0d76c-0b82-4c94-aa66-bc08d783d330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:24:06.509282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:24:06.566335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 750/1518] 10.222.0.71 () {58 vars in 1015 bytes} [Tue Aug 30 04:24:06 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:24:13.304360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=56 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:24:14.775558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcb0233-dd58-41e8-8716-8050014862f5 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:14.776200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcb0233-dd58-41e8-8716-8050014862f5 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:14.776200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcb0233-dd58-41e8-8716-8050014862f5 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:14.777498 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 769/1519] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 04:24:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:24:14.782482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967a8845-5ce3-424e-8ab6-9f4415664fe5 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:14.782916 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967a8845-5ce3-424e-8ab6-9f4415664fe5 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:14.783016 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967a8845-5ce3-424e-8ab6-9f4415664fe5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:14.806614 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-967a8845-5ce3-424e-8ab6-9f4415664fe5 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:24:14.807834 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-967a8845-5ce3-424e-8ab6-9f4415664fe5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:24:14.815682 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 751/1520] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 04:24:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:24:14.972487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcb0233-dd58-41e8-8716-8050014862f5 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:24:14.987990 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcb0233-dd58-41e8-8716-8050014862f5 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:24:14.993362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988820f9-5a1f-44a5-83d4-aee06797c35a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:24:14.993947 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988820f9-5a1f-44a5-83d4-aee06797c35a service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:14.994220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988820f9-5a1f-44a5-83d4-aee06797c35a service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:14.994484 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988820f9-5a1f-44a5-83d4-aee06797c35a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:15.002766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988820f9-5a1f-44a5-83d4-aee06797c35a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:24:15.004347 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988820f9-5a1f-44a5-83d4-aee06797c35a service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:24:15.014212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 770/1521] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:24:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:24:15.838250 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e8745d-9b15-474f-ac6d-0e8d7bafd552 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:15.838807 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e8745d-9b15-474f-ac6d-0e8d7bafd552 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:15.838807 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e8745d-9b15-474f-ac6d-0e8d7bafd552 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:15.840076 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 752/1522] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 04:24:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:24:15.844963 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936c4f6a-3472-4c93-88f0-f89fda1f7c2d None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:15.845475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936c4f6a-3472-4c93-88f0-f89fda1f7c2d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:15.845582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936c4f6a-3472-4c93-88f0-f89fda1f7c2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:15.865105 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-936c4f6a-3472-4c93-88f0-f89fda1f7c2d None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:24:15.866804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-936c4f6a-3472-4c93-88f0-f89fda1f7c2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:24:15.874934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 771/1523] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 04:24:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:24:16.027417 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e8745d-9b15-474f-ac6d-0e8d7bafd552 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:24:16.042708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e8745d-9b15-474f-ac6d-0e8d7bafd552 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:24:16.047992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524e29ed-740b-4409-ae47-0ff625d7afbd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:24:16.048579 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524e29ed-740b-4409-ae47-0ff625d7afbd service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:16.048856 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524e29ed-740b-4409-ae47-0ff625d7afbd service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:16.049156 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524e29ed-740b-4409-ae47-0ff625d7afbd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:16.057294 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524e29ed-740b-4409-ae47-0ff625d7afbd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:24:16.058901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524e29ed-740b-4409-ae47-0ff625d7afbd service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:24:16.068247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 753/1524] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:24:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:24:18.423673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a2927d-3628-4c11-95ac-d333616445d5 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:18.424205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a2927d-3628-4c11-95ac-d333616445d5 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:18.424205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a2927d-3628-4c11-95ac-d333616445d5 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:18.425504 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 772/1525] 10.222.0.85 () {54 vars in 961 bytes} [Tue Aug 30 04:24:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:24:18.430536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8748e6-8c3d-4610-b954-c7667373d6d3 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:18.431029 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8748e6-8c3d-4610-b954-c7667373d6d3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:18.431113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8748e6-8c3d-4610-b954-c7667373d6d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:18.450020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3b8748e6-8c3d-4610-b954-c7667373d6d3 None None] MFA Rules not processed for user `359d79cbfcad49beba2b495066824128`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:24:18.451224 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3b8748e6-8c3d-4610-b954-c7667373d6d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:24:18.460418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 754/1526] 10.222.0.85 () {58 vars in 1058 bytes} [Tue Aug 30 04:24:18 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:24:18.467968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a2927d-3628-4c11-95ac-d333616445d5 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:24:18.482160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a2927d-3628-4c11-95ac-d333616445d5 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:24:18.487439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79abb51-443f-4dfd-8586-3dae56ab4d6b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '359d79cbfcad49beba2b495066824128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:24:18.488048 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79abb51-443f-4dfd-8586-3dae56ab4d6b service placement] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:18.488320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79abb51-443f-4dfd-8586-3dae56ab4d6b service placement] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:18.488589 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79abb51-443f-4dfd-8586-3dae56ab4d6b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:18.497446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79abb51-443f-4dfd-8586-3dae56ab4d6b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:24:18.499041 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79abb51-443f-4dfd-8586-3dae56ab4d6b service placement] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:24:18.509146 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 773/1527] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:24:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:24:27.313606 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e651a2-71ea-4317-889d-61c9d06be479 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:27.314167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e651a2-71ea-4317-889d-61c9d06be479 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:27.314167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e651a2-71ea-4317-889d-61c9d06be479 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:27.315505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 755/1528] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 04:24:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:24:27.320735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ee762c-5c3a-430c-9d78-349a1bf1cad1 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:27.321158 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ee762c-5c3a-430c-9d78-349a1bf1cad1 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:27.321303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ee762c-5c3a-430c-9d78-349a1bf1cad1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:27.340419 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-57ee762c-5c3a-430c-9d78-349a1bf1cad1 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:24:27.341746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-57ee762c-5c3a-430c-9d78-349a1bf1cad1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:24:27.349423 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 774/1529] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 04:24:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:24:27.507166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e651a2-71ea-4317-889d-61c9d06be479 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:24:27.521423 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e651a2-71ea-4317-889d-61c9d06be479 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:24:27.527519 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cb2648-055f-4d78-9b85-36647d159c7b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:24:27.528111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cb2648-055f-4d78-9b85-36647d159c7b service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:27.528396 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cb2648-055f-4d78-9b85-36647d159c7b service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:27.528671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cb2648-055f-4d78-9b85-36647d159c7b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:27.536733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cb2648-055f-4d78-9b85-36647d159c7b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:24:27.538858 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cb2648-055f-4d78-9b85-36647d159c7b service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:24:27.548166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 756/1530] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:24:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:24:28.384736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095c0f00-803b-4b6b-816d-0454b8efb066 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:28.385295 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095c0f00-803b-4b6b-816d-0454b8efb066 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:28.385295 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095c0f00-803b-4b6b-816d-0454b8efb066 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:28.386570 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 775/1531] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 04:24:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:24:28.391451 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c4541b-02d4-4137-9114-8928deb6228e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:28.391921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c4541b-02d4-4137-9114-8928deb6228e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:28.392018 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c4541b-02d4-4137-9114-8928deb6228e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:28.411946 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-30c4541b-02d4-4137-9114-8928deb6228e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:24:28.413195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-30c4541b-02d4-4137-9114-8928deb6228e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:24:28.421079 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 757/1532] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 04:24:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:24:28.573970 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095c0f00-803b-4b6b-816d-0454b8efb066 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:24:28.589094 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095c0f00-803b-4b6b-816d-0454b8efb066 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:24:28.594463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00431fa-cecd-478c-a0c9-0fcf73006e8f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:24:28.595051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00431fa-cecd-478c-a0c9-0fcf73006e8f service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:28.595331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00431fa-cecd-478c-a0c9-0fcf73006e8f service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:28.595600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00431fa-cecd-478c-a0c9-0fcf73006e8f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:28.603634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00431fa-cecd-478c-a0c9-0fcf73006e8f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:24:28.605226 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00431fa-cecd-478c-a0c9-0fcf73006e8f service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:24:28.615002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 776/1533] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:24:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:24:32.482438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d05c3b-ecb7-48ff-87f5-99c6f2ed5b19 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:32.483010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d05c3b-ecb7-48ff-87f5-99c6f2ed5b19 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:32.483010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d05c3b-ecb7-48ff-87f5-99c6f2ed5b19 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:32.484344 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 758/1534] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 04:24:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:24:32.489717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8347119-01de-4a54-b868-c0fe00015a7f None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:32.490163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8347119-01de-4a54-b868-c0fe00015a7f None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:32.490800 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8347119-01de-4a54-b868-c0fe00015a7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:32.511109 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-a8347119-01de-4a54-b868-c0fe00015a7f None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:24:32.512312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-a8347119-01de-4a54-b868-c0fe00015a7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:24:32.520379 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 777/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 04:24:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:24:32.804999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d05c3b-ecb7-48ff-87f5-99c6f2ed5b19 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:24:32.806741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d7d05c3b-ecb7-48ff-87f5-99c6f2ed5b19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:24:32.886050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d05c3b-ecb7-48ff-87f5-99c6f2ed5b19 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:24:32.889564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1055ed-b142-4ec8-9297-833ba03d52b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:24:32.890169 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1055ed-b142-4ec8-9297-833ba03d52b0 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:32.890448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1055ed-b142-4ec8-9297-833ba03d52b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:32.890720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1055ed-b142-4ec8-9297-833ba03d52b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:32.899927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1055ed-b142-4ec8-9297-833ba03d52b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:24:32.901419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1055ed-b142-4ec8-9297-833ba03d52b0 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:24:32.911028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 759/1536] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:24:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:24:34.821397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51ed8ea-b037-4e14-af01-c5ebe1f685ad None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:34.821963 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51ed8ea-b037-4e14-af01-c5ebe1f685ad None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:34.821963 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51ed8ea-b037-4e14-af01-c5ebe1f685ad None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:34.823217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 778/1537] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 04:24:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:24:34.828624 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867fc02c-3f13-4374-98e2-c58f3bb5cf0b None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:34.829166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867fc02c-3f13-4374-98e2-c58f3bb5cf0b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:34.829166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867fc02c-3f13-4374-98e2-c58f3bb5cf0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:34.848365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-867fc02c-3f13-4374-98e2-c58f3bb5cf0b None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:24:34.849639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-867fc02c-3f13-4374-98e2-c58f3bb5cf0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:24:34.857542 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 760/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 04:24:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:24:35.148860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51ed8ea-b037-4e14-af01-c5ebe1f685ad None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:24:35.160009 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51ed8ea-b037-4e14-af01-c5ebe1f685ad None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:24:35.163698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401299ac-afd6-4a9e-878f-7527d77b1489 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:24:35.164291 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401299ac-afd6-4a9e-878f-7527d77b1489 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:35.164599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401299ac-afd6-4a9e-878f-7527d77b1489 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:35.164865 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401299ac-afd6-4a9e-878f-7527d77b1489 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:35.173744 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401299ac-afd6-4a9e-878f-7527d77b1489 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:24:35.175169 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401299ac-afd6-4a9e-878f-7527d77b1489 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:24:35.184774 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 779/1539] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:24:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:24:38.723195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec5e957-e9d6-42de-ae90-8619c424deea None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:38.723818 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec5e957-e9d6-42de-ae90-8619c424deea None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:38.724235 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec5e957-e9d6-42de-ae90-8619c424deea None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:38.726119 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 761/1540] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 04:24:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:24:38.731618 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d54139-08de-4434-b059-f708b7d75ad4 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:38.732072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d54139-08de-4434-b059-f708b7d75ad4 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:38.732175 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d54139-08de-4434-b059-f708b7d75ad4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:38.752265 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-56d54139-08de-4434-b059-f708b7d75ad4 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:24:38.753511 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-56d54139-08de-4434-b059-f708b7d75ad4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:24:38.761295 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 780/1541] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 04:24:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:24:39.320089 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec5e957-e9d6-42de-ae90-8619c424deea None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:24:39.321770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-aec5e957-e9d6-42de-ae90-8619c424deea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:24:39.325983 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=83 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:24:39.400285 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec5e957-e9d6-42de-ae90-8619c424deea None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:24:39.403791 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0447372-e513-44da-8c71-7942e0c3f4df service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:24:39.404401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0447372-e513-44da-8c71-7942e0c3f4df service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:39.404680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0447372-e513-44da-8c71-7942e0c3f4df service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:39.404953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0447372-e513-44da-8c71-7942e0c3f4df service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:39.413752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0447372-e513-44da-8c71-7942e0c3f4df service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:24:39.415183 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0447372-e513-44da-8c71-7942e0c3f4df service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:24:39.424496 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 762/1542] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:24:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:24:41.346462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44892d7f-f25b-4927-bef4-5d991da6bdf2 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:41.346991 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44892d7f-f25b-4927-bef4-5d991da6bdf2 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:41.346991 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44892d7f-f25b-4927-bef4-5d991da6bdf2 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:41.348329 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 781/1543] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 04:24:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:24:41.353785 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb5de8e-0e51-4a60-9b16-e91aad4b50a1 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:41.354173 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb5de8e-0e51-4a60-9b16-e91aad4b50a1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:41.354341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb5de8e-0e51-4a60-9b16-e91aad4b50a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:41.374492 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-6cb5de8e-0e51-4a60-9b16-e91aad4b50a1 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:24:41.375687 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6cb5de8e-0e51-4a60-9b16-e91aad4b50a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:24:41.384077 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 763/1544] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 04:24:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:24:41.934376 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44892d7f-f25b-4927-bef4-5d991da6bdf2 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:24:41.936556 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-44892d7f-f25b-4927-bef4-5d991da6bdf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:24:41.950367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44892d7f-f25b-4927-bef4-5d991da6bdf2 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:24:41.954008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2496b64c-947e-4d9b-9652-da254e4e7ffe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:24:41.954617 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2496b64c-947e-4d9b-9652-da254e4e7ffe service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:24:41.954901 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2496b64c-947e-4d9b-9652-da254e4e7ffe service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:24:41.955175 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2496b64c-947e-4d9b-9652-da254e4e7ffe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:24:41.964826 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2496b64c-947e-4d9b-9652-da254e4e7ffe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:24:41.966288 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2496b64c-947e-4d9b-9652-da254e4e7ffe service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:24:41.976126 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 782/1545] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:24:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:24:51.369446 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=37 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:24:51.971506 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=42 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:08.498310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588298b7-429d-4de0-8431-4d0818081dcb None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:08.499819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588298b7-429d-4de0-8431-4d0818081dcb None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:08.499819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588298b7-429d-4de0-8431-4d0818081dcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:08.526988 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-588298b7-429d-4de0-8431-4d0818081dcb None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:08.528507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-588298b7-429d-4de0-8431-4d0818081dcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:08.541611 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 764/1546] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:08.550554 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7be30a1-35bb-43d2-a75b-45f4b2be3e62 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:08.551630 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7be30a1-35bb-43d2-a75b-45f4b2be3e62 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:08.552140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7be30a1-35bb-43d2-a75b-45f4b2be3e62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:08.580021 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-d7be30a1-35bb-43d2-a75b-45f4b2be3e62 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:08.581572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d7be30a1-35bb-43d2-a75b-45f4b2be3e62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:08.594616 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 783/1547] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:08.599383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588298b7-429d-4de0-8431-4d0818081dcb None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:08.615664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588298b7-429d-4de0-8431-4d0818081dcb None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:08.628490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14df7b9e-aed8-43fa-ba0c-b3d08ab2f6bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:08.630059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14df7b9e-aed8-43fa-ba0c-b3d08ab2f6bd admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:08.630345 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14df7b9e-aed8-43fa-ba0c-b3d08ab2f6bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:08.630615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14df7b9e-aed8-43fa-ba0c-b3d08ab2f6bd admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:08.631299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14df7b9e-aed8-43fa-ba0c-b3d08ab2f6bd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:08.631600 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14df7b9e-aed8-43fa-ba0c-b3d08ab2f6bd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:08.633294 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14df7b9e-aed8-43fa-ba0c-b3d08ab2f6bd admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:08.653627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 765/1548] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:26:08.662463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7be30a1-35bb-43d2-a75b-45f4b2be3e62 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:08.678320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7be30a1-35bb-43d2-a75b-45f4b2be3e62 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:08.683126 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fa00be-e724-41b0-97e9-bf91f0206343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:08.683736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fa00be-e724-41b0-97e9-bf91f0206343 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:08.684021 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fa00be-e724-41b0-97e9-bf91f0206343 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:08.684294 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fa00be-e724-41b0-97e9-bf91f0206343 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:08.684925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fa00be-e724-41b0-97e9-bf91f0206343 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:08.687058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fa00be-e724-41b0-97e9-bf91f0206343 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:08.704322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0604d6-1aaf-43f8-859c-84ad538acbeb None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:08.704663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0604d6-1aaf-43f8-859c-84ad538acbeb None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:08.704953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0604d6-1aaf-43f8-859c-84ad538acbeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:08.741452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-9a0604d6-1aaf-43f8-859c-84ad538acbeb None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:08.742337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 784/1549] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:08 2022] POST /identity/v3/projects => generated 393 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:08.742646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9a0604d6-1aaf-43f8-859c-84ad538acbeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:08.749794 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fa00be-e724-41b0-97e9-bf91f0206343 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:08.851642 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 766/1550] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:26:08.852030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fa00be-e724-41b0-97e9-bf91f0206343 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:08.857456 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc755c8-1fc1-41db-9742-47050a5667e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:08.858090 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc755c8-1fc1-41db-9742-47050a5667e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:08.858373 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc755c8-1fc1-41db-9742-47050a5667e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:08.858649 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc755c8-1fc1-41db-9742-47050a5667e4 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:08.859305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc755c8-1fc1-41db-9742-47050a5667e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:08.860365 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: WARNING py.warnings [None req-1cc755c8-1fc1-41db-9742-47050a5667e4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 04:26:08.860365 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: warnings.warn(msg) Aug 30 04:26:08.860365 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: Aug 30 04:26:08.861970 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc755c8-1fc1-41db-9742-47050a5667e4 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:08.866580 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d313af6-4d4d-475e-8a91-4f12a5a07983 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:08.866861 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d313af6-4d4d-475e-8a91-4f12a5a07983 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:08.867130 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d313af6-4d4d-475e-8a91-4f12a5a07983 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:08.894015 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 785/1551] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:08 2022] POST /identity/v3/users => generated 297 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:08.902294 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3d313af6-4d4d-475e-8a91-4f12a5a07983 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:08.902537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc755c8-1fc1-41db-9742-47050a5667e4 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:08.903527 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3d313af6-4d4d-475e-8a91-4f12a5a07983 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:08.914298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc755c8-1fc1-41db-9742-47050a5667e4 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:08.922733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 767/1552] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:26:08.923040 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7d5117-475a-418e-b7d1-b33496ec534e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:08.923821 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7d5117-475a-418e-b7d1-b33496ec534e admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:08.924109 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7d5117-475a-418e-b7d1-b33496ec534e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:08.924386 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7d5117-475a-418e-b7d1-b33496ec534e admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:08.925066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7d5117-475a-418e-b7d1-b33496ec534e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:08.926765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7d5117-475a-418e-b7d1-b33496ec534e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:08.929131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d313af6-4d4d-475e-8a91-4f12a5a07983 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:08.939077 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d313af6-4d4d-475e-8a91-4f12a5a07983 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:08.944576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 786/1553] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:08 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:08.948772 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7d5117-475a-418e-b7d1-b33496ec534e admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:08.949557 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87ea203-2830-4f7e-825b-31d36883fdff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:08.950197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87ea203-2830-4f7e-825b-31d36883fdff admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:08.950472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87ea203-2830-4f7e-825b-31d36883fdff admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:08.950740 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87ea203-2830-4f7e-825b-31d36883fdff admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:08.951398 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87ea203-2830-4f7e-825b-31d36883fdff admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:08.951694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87ea203-2830-4f7e-825b-31d36883fdff admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:08.953346 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87ea203-2830-4f7e-825b-31d36883fdff admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:08.963276 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7d5117-475a-418e-b7d1-b33496ec534e admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:08.967820 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 768/1554] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:26:08.974271 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509e8816-7500-49e1-9cf5-a0e8322237b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:08.974907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509e8816-7500-49e1-9cf5-a0e8322237b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:08.975191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509e8816-7500-49e1-9cf5-a0e8322237b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:08.975466 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509e8816-7500-49e1-9cf5-a0e8322237b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:08.976122 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509e8816-7500-49e1-9cf5-a0e8322237b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:08.977764 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509e8816-7500-49e1-9cf5-a0e8322237b2 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:08.980477 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87ea203-2830-4f7e-825b-31d36883fdff admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:08.995571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 787/1555] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:08 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:26:08.995811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87ea203-2830-4f7e-825b-31d36883fdff admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:09.001254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0667b20-0d74-4f84-a2c0-758860f6229a None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.001581 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0667b20-0d74-4f84-a2c0-758860f6229a None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.001773 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0667b20-0d74-4f84-a2c0-758860f6229a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.003278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cecbb64-0773-4308-b5cf-789df254d107 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:09.003939 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cecbb64-0773-4308-b5cf-789df254d107 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.004266 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cecbb64-0773-4308-b5cf-789df254d107 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.004542 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cecbb64-0773-4308-b5cf-789df254d107 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.005222 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cecbb64-0773-4308-b5cf-789df254d107 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:09.006590 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: WARNING py.warnings [None req-3cecbb64-0773-4308-b5cf-789df254d107 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 04:26:09.006590 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: warnings.warn(msg) Aug 30 04:26:09.006590 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: Aug 30 04:26:09.007796 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cecbb64-0773-4308-b5cf-789df254d107 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:09.046535 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-c0667b20-0d74-4f84-a2c0-758860f6229a None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:09.047785 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c0667b20-0d74-4f84-a2c0-758860f6229a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:09.055073 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 769/1556] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:08 2022] POST /identity/v3/projects => generated 375 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:09.055735 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cecbb64-0773-4308-b5cf-789df254d107 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:09.061550 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 788/1557] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:26:09.062200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0667b20-0d74-4f84-a2c0-758860f6229a None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:09.078050 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0667b20-0d74-4f84-a2c0-758860f6229a None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:09.189828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c06acf-3865-40b2-b352-dace0c465018 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:09.190463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c06acf-3865-40b2-b352-dace0c465018 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.190743 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c06acf-3865-40b2-b352-dace0c465018 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.191018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c06acf-3865-40b2-b352-dace0c465018 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.191712 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c06acf-3865-40b2-b352-dace0c465018 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:09.193890 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c06acf-3865-40b2-b352-dace0c465018 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:09.219283 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cecbb64-0773-4308-b5cf-789df254d107 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:09.232312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 789/1558] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:09 2022] POST /identity/v3/users => generated 288 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:09.233334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320af117-5813-41aa-a023-7545849ee33e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:09.233498 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf820fa-c4d5-49c4-b69a-b143d33917b7 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.233738 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf820fa-c4d5-49c4-b69a-b143d33917b7 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.234012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf820fa-c4d5-49c4-b69a-b143d33917b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.234841 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320af117-5813-41aa-a023-7545849ee33e admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.235716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320af117-5813-41aa-a023-7545849ee33e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.236215 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320af117-5813-41aa-a023-7545849ee33e admin admin] PATH_INFO: `/v3/projects/edc09fe630034be4b292562e2de6e3f9/users/cbf574ed57d446bc8fabe7813e64e92c/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.264253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-aaf820fa-c4d5-49c4-b69a-b143d33917b7 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:09.265420 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-aaf820fa-c4d5-49c4-b69a-b143d33917b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:09.269212 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320af117-5813-41aa-a023-7545849ee33e admin admin] RBAC: Authorizing `identity:create_grant(project_id=edc09fe630034be4b292562e2de6e3f9, user_id=cbf574ed57d446bc8fabe7813e64e92c, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:09.270692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: WARNING py.warnings [None req-320af117-5813-41aa-a023-7545849ee33e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 04:26:09.270692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: warnings.warn(msg) Aug 30 04:26:09.270692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: Aug 30 04:26:09.272636 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320af117-5813-41aa-a023-7545849ee33e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:09.282849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 790/1559] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:09.283895 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59444549-1e9a-411d-a16d-09372bb4bbf7 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.284178 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59444549-1e9a-411d-a16d-09372bb4bbf7 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.284448 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59444549-1e9a-411d-a16d-09372bb4bbf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.295842 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 770/1560] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:09 2022] PUT /identity/v3/projects/edc09fe630034be4b292562e2de6e3f9/users/cbf574ed57d446bc8fabe7813e64e92c/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:09.296687 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320af117-5813-41aa-a023-7545849ee33e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:09.314172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:09.327652 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-59444549-1e9a-411d-a16d-09372bb4bbf7 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:09.329025 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-59444549-1e9a-411d-a16d-09372bb4bbf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:09.342945 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:09.392082 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320af117-5813-41aa-a023-7545849ee33e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:09.396791 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0cb233-2239-4a85-90fc-442b17212858 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:09.397433 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0cb233-2239-4a85-90fc-442b17212858 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.397714 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0cb233-2239-4a85-90fc-442b17212858 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.397985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0cb233-2239-4a85-90fc-442b17212858 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.398623 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0cb233-2239-4a85-90fc-442b17212858 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:09.400236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0cb233-2239-4a85-90fc-442b17212858 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:09.410690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 771/1561] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:09 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:09.417074 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 791/1562] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:09.417253 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9723e6-2b62-43e7-9797-c1ae88737ab6 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.417253 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9723e6-2b62-43e7-9797-c1ae88737ab6 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.417437 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59444549-1e9a-411d-a16d-09372bb4bbf7 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:09.417534 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9723e6-2b62-43e7-9797-c1ae88737ab6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.423415 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59444549-1e9a-411d-a16d-09372bb4bbf7 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:09.428694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5990b01-9ef2-4e38-b156-2a9bd1ec4d5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:09.429335 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5990b01-9ef2-4e38-b156-2a9bd1ec4d5d admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.429615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5990b01-9ef2-4e38-b156-2a9bd1ec4d5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.429887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5990b01-9ef2-4e38-b156-2a9bd1ec4d5d admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.430535 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5990b01-9ef2-4e38-b156-2a9bd1ec4d5d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:09.430840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5990b01-9ef2-4e38-b156-2a9bd1ec4d5d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:09.431856 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: WARNING py.warnings [None req-d5990b01-9ef2-4e38-b156-2a9bd1ec4d5d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 30 04:26:09.431856 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: warnings.warn(msg) Aug 30 04:26:09.431856 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: Aug 30 04:26:09.432747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5990b01-9ef2-4e38-b156-2a9bd1ec4d5d admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:09.433634 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-aa9723e6-2b62-43e7-9797-c1ae88737ab6 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:09.435099 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-aa9723e6-2b62-43e7-9797-c1ae88737ab6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:09.443792 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 772/1563] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:09.445168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 792/1564] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:26:09.446306 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c987201-b670-473f-82da-aebefb0c0999 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.446598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c987201-b670-473f-82da-aebefb0c0999 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.446873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c987201-b670-473f-82da-aebefb0c0999 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.449127 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d809ac6-8549-4982-9b92-6ed2e3e2ac43 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.449681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d809ac6-8549-4982-9b92-6ed2e3e2ac43 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.450021 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d809ac6-8549-4982-9b92-6ed2e3e2ac43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.468162 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-8c987201-b670-473f-82da-aebefb0c0999 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:09.469295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-0d809ac6-8549-4982-9b92-6ed2e3e2ac43 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:09.469431 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8c987201-b670-473f-82da-aebefb0c0999 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:09.470451 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0d809ac6-8549-4982-9b92-6ed2e3e2ac43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:09.485530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 793/1565] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:09.486295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 773/1566] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:09.486295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2265c2d4-8e7e-405b-8dba-9d9cfeeb5436 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.486295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2265c2d4-8e7e-405b-8dba-9d9cfeeb5436 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.486295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2265c2d4-8e7e-405b-8dba-9d9cfeeb5436 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.486660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041b90d1-c5bb-444b-b1d1-091c89666e7a None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.486660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041b90d1-c5bb-444b-b1d1-091c89666e7a None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.486660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041b90d1-c5bb-444b-b1d1-091c89666e7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.505458 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-041b90d1-c5bb-444b-b1d1-091c89666e7a None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:09.506115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-2265c2d4-8e7e-405b-8dba-9d9cfeeb5436 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:09.506671 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-041b90d1-c5bb-444b-b1d1-091c89666e7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:09.507308 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2265c2d4-8e7e-405b-8dba-9d9cfeeb5436 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:09.515174 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 794/1567] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:09.515314 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 774/1568] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:09.516287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6a71b4-92f5-47eb-8141-73b5138ab2e6 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.516437 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041b90d1-c5bb-444b-b1d1-091c89666e7a None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:09.516563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6a71b4-92f5-47eb-8141-73b5138ab2e6 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.516825 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6a71b4-92f5-47eb-8141-73b5138ab2e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.527043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041b90d1-c5bb-444b-b1d1-091c89666e7a None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:09.533324 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94c5876-abf5-42f7-9e89-c979d4feab9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:09.533936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94c5876-abf5-42f7-9e89-c979d4feab9e admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.534216 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94c5876-abf5-42f7-9e89-c979d4feab9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.534490 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94c5876-abf5-42f7-9e89-c979d4feab9e admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.535654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94c5876-abf5-42f7-9e89-c979d4feab9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:09.537351 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94c5876-abf5-42f7-9e89-c979d4feab9e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:09.542197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-da6a71b4-92f5-47eb-8141-73b5138ab2e6 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:09.543401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-da6a71b4-92f5-47eb-8141-73b5138ab2e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:09.547489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 795/1569] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:09 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:09.548679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4da380-cf7a-4a64-aaa7-0a0fd0b5da03 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.548966 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4da380-cf7a-4a64-aaa7-0a0fd0b5da03 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.549306 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4da380-cf7a-4a64-aaa7-0a0fd0b5da03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.551850 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 775/1570] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:09.553863 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6a71b4-92f5-47eb-8141-73b5138ab2e6 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:09.563870 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6a71b4-92f5-47eb-8141-73b5138ab2e6 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:09.568529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279e04d6-29ee-4ef7-acf7-bf66b2a4c06d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:09.569163 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279e04d6-29ee-4ef7-acf7-bf66b2a4c06d admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.569430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279e04d6-29ee-4ef7-acf7-bf66b2a4c06d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.569701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279e04d6-29ee-4ef7-acf7-bf66b2a4c06d admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.569941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-5a4da380-cf7a-4a64-aaa7-0a0fd0b5da03 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:09.570454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279e04d6-29ee-4ef7-acf7-bf66b2a4c06d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:09.571158 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-5a4da380-cf7a-4a64-aaa7-0a0fd0b5da03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:09.572249 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279e04d6-29ee-4ef7-acf7-bf66b2a4c06d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:09.580143 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 796/1571] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:09.581290 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db69518-d8b9-4421-8de9-576f81b1f985 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.581578 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db69518-d8b9-4421-8de9-576f81b1f985 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.581845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db69518-d8b9-4421-8de9-576f81b1f985 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.603870 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 776/1572] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:09 2022] POST /identity/v3/projects => generated 403 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:09.603870 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279e04d6-29ee-4ef7-acf7-bf66b2a4c06d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:09.604172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-6db69518-d8b9-4421-8de9-576f81b1f985 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:09.604172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-6db69518-d8b9-4421-8de9-576f81b1f985 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:09.656715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279e04d6-29ee-4ef7-acf7-bf66b2a4c06d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:09.661086 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b521ef-d3de-4ee9-963f-302784b27aae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:09.661761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b521ef-d3de-4ee9-963f-302784b27aae service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.662058 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b521ef-d3de-4ee9-963f-302784b27aae service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.662333 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b521ef-d3de-4ee9-963f-302784b27aae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.672607 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 797/1573] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:09.673718 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bef10c-4b9d-4405-95cf-2de4325a4546 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.674077 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bef10c-4b9d-4405-95cf-2de4325a4546 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.674434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bef10c-4b9d-4405-95cf-2de4325a4546 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.694669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-93bef10c-4b9d-4405-95cf-2de4325a4546 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:09.695879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-93bef10c-4b9d-4405-95cf-2de4325a4546 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:09.703808 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 798/1574] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:09.707604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d55f616-c521-470f-aa98-3db2f6d45fe2 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.707604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d55f616-c521-470f-aa98-3db2f6d45fe2 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.707604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d55f616-c521-470f-aa98-3db2f6d45fe2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.724591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b521ef-d3de-4ee9-963f-302784b27aae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:09.724756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-3d55f616-c521-470f-aa98-3db2f6d45fe2 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:09.725932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-3d55f616-c521-470f-aa98-3db2f6d45fe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:09.726084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b521ef-d3de-4ee9-963f-302784b27aae service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:09.733911 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 799/1575] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:09.734749 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d55f616-c521-470f-aa98-3db2f6d45fe2 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:09.735547 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 777/1576] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:09.736235 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b521ef-d3de-4ee9-963f-302784b27aae service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:09.744804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d55f616-c521-470f-aa98-3db2f6d45fe2 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:09.745920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b521ef-d3de-4ee9-963f-302784b27aae service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:09.750166 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e86cec5-5df1-44cd-b4f5-5ee9e825ae5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:09.750567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e86cec5-5df1-44cd-b4f5-5ee9e825ae5f admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.750847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e86cec5-5df1-44cd-b4f5-5ee9e825ae5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.751121 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e86cec5-5df1-44cd-b4f5-5ee9e825ae5f admin admin] PATH_INFO: `/v3/projects/b1363dbf0d3d4afe8d1b407483d3c6fe/users/37df071b935f4a51b81e5130d9c44e1f/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.752645 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaf8435-7e2a-43c7-8135-d9f8f9f60cd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:09.753279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaf8435-7e2a-43c7-8135-d9f8f9f60cd3 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.753560 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaf8435-7e2a-43c7-8135-d9f8f9f60cd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.753860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaf8435-7e2a-43c7-8135-d9f8f9f60cd3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.754507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaf8435-7e2a-43c7-8135-d9f8f9f60cd3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:09.754808 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaf8435-7e2a-43c7-8135-d9f8f9f60cd3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:09.756407 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaf8435-7e2a-43c7-8135-d9f8f9f60cd3 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:09.767149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e86cec5-5df1-44cd-b4f5-5ee9e825ae5f admin admin] RBAC: Authorizing `identity:create_grant(project_id=b1363dbf0d3d4afe8d1b407483d3c6fe, user_id=37df071b935f4a51b81e5130d9c44e1f, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:09.769268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 778/1577] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:26:09.770336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e86cec5-5df1-44cd-b4f5-5ee9e825ae5f admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:09.774307 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaf8435-7e2a-43c7-8135-d9f8f9f60cd3 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:09.793299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 800/1578] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:09 2022] PUT /identity/v3/projects/b1363dbf0d3d4afe8d1b407483d3c6fe/users/37df071b935f4a51b81e5130d9c44e1f/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:09.793506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaf8435-7e2a-43c7-8135-d9f8f9f60cd3 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:09.793609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e86cec5-5df1-44cd-b4f5-5ee9e825ae5f admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:09.799537 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:09.833691 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:09.871418 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e86cec5-5df1-44cd-b4f5-5ee9e825ae5f admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:09.873471 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c5f988-6f02-432c-bd2d-f8a1fb7fdde4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:09.874334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c5f988-6f02-432c-bd2d-f8a1fb7fdde4 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.874755 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c5f988-6f02-432c-bd2d-f8a1fb7fdde4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.875036 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c5f988-6f02-432c-bd2d-f8a1fb7fdde4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.875684 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c5f988-6f02-432c-bd2d-f8a1fb7fdde4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:09.875982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c5f988-6f02-432c-bd2d-f8a1fb7fdde4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:09.877732 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c5f988-6f02-432c-bd2d-f8a1fb7fdde4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:09.878973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ff143e-08e1-4177-b59f-90d490f24de8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:09.879834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ff143e-08e1-4177-b59f-90d490f24de8 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.880303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ff143e-08e1-4177-b59f-90d490f24de8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.880766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ff143e-08e1-4177-b59f-90d490f24de8 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.881438 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ff143e-08e1-4177-b59f-90d490f24de8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:09.883532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ff143e-08e1-4177-b59f-90d490f24de8 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:09.891811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 779/1579] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:26:09.892781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c5f988-6f02-432c-bd2d-f8a1fb7fdde4 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:09.904829 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c5f988-6f02-432c-bd2d-f8a1fb7fdde4 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:09.911159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52c0af5-a201-4c13-b439-3a8e64ea9a62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:09.914539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 801/1580] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:09 2022] POST /identity/v3/users => generated 302 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:09.914539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ff143e-08e1-4177-b59f-90d490f24de8 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:09.914741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52c0af5-a201-4c13-b439-3a8e64ea9a62 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.914741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52c0af5-a201-4c13-b439-3a8e64ea9a62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.914741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52c0af5-a201-4c13-b439-3a8e64ea9a62 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.914741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52c0af5-a201-4c13-b439-3a8e64ea9a62 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:09.914741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52c0af5-a201-4c13-b439-3a8e64ea9a62 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:09.916622 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52c0af5-a201-4c13-b439-3a8e64ea9a62 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:09.924859 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ff143e-08e1-4177-b59f-90d490f24de8 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:09.930786 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 780/1581] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:26:09.931709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52c0af5-a201-4c13-b439-3a8e64ea9a62 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:09.938657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e8b1ed-fc49-4940-8177-d88a42b4e266 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:09.939451 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e8b1ed-fc49-4940-8177-d88a42b4e266 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.939777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e8b1ed-fc49-4940-8177-d88a42b4e266 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.940131 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e8b1ed-fc49-4940-8177-d88a42b4e266 admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.940878 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e8b1ed-fc49-4940-8177-d88a42b4e266 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:09.941237 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e8b1ed-fc49-4940-8177-d88a42b4e266 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:09.942972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e8b1ed-fc49-4940-8177-d88a42b4e266 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:09.949133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52c0af5-a201-4c13-b439-3a8e64ea9a62 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:09.954005 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b63f5c4-a9f5-4909-9c53-7dd3d928182d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:09.954875 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b63f5c4-a9f5-4909-9c53-7dd3d928182d admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.955554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b63f5c4-a9f5-4909-9c53-7dd3d928182d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.955827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b63f5c4-a9f5-4909-9c53-7dd3d928182d admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.956423 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b63f5c4-a9f5-4909-9c53-7dd3d928182d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:09.956788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b63f5c4-a9f5-4909-9c53-7dd3d928182d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:09.959137 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b63f5c4-a9f5-4909-9c53-7dd3d928182d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:09.969830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 802/1582] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:26:09.971261 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e8b1ed-fc49-4940-8177-d88a42b4e266 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:09.974458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 781/1583] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:26:09.975820 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e94900-76a1-473d-9356-89490afbd6fd None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.976111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e94900-76a1-473d-9356-89490afbd6fd None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.976389 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e94900-76a1-473d-9356-89490afbd6fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.987480 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e8b1ed-fc49-4940-8177-d88a42b4e266 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:09.992430 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88b5e0f-701d-4ea0-81cc-085aaaded8ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:09.993142 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88b5e0f-701d-4ea0-81cc-085aaaded8ee admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:09.993487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88b5e0f-701d-4ea0-81cc-085aaaded8ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:09.993825 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88b5e0f-701d-4ea0-81cc-085aaaded8ee admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:09.994515 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88b5e0f-701d-4ea0-81cc-085aaaded8ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:09.996755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88b5e0f-701d-4ea0-81cc-085aaaded8ee admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:10.007120 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-45e94900-76a1-473d-9356-89490afbd6fd None None] MFA Rules not processed for user `37df071b935f4a51b81e5130d9c44e1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:10.008357 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-45e94900-76a1-473d-9356-89490afbd6fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:10.037614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 803/1584] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:09 2022] POST /identity/v3/projects => generated 371 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:10.038715 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88b5e0f-701d-4ea0-81cc-085aaaded8ee admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:10.087962 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 782/1585] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:09 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:10.089117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e94900-76a1-473d-9356-89490afbd6fd None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:10.122585 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88b5e0f-701d-4ea0-81cc-085aaaded8ee admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:10.126610 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e94900-76a1-473d-9356-89490afbd6fd None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:10.127915 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c7bfda-618d-4a8d-b352-a3519b41c3bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:10.128705 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c7bfda-618d-4a8d-b352-a3519b41c3bd admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.129087 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c7bfda-618d-4a8d-b352-a3519b41c3bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.129470 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c7bfda-618d-4a8d-b352-a3519b41c3bd admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.130626 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c7bfda-618d-4a8d-b352-a3519b41c3bd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:10.133271 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c7bfda-618d-4a8d-b352-a3519b41c3bd admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:10.142419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1dd171-4cdb-48d9-88bd-754ce8434828 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:10.143640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1dd171-4cdb-48d9-88bd-754ce8434828 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.144093 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1dd171-4cdb-48d9-88bd-754ce8434828 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.145192 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1dd171-4cdb-48d9-88bd-754ce8434828 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.146041 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1dd171-4cdb-48d9-88bd-754ce8434828 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:10.147825 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1dd171-4cdb-48d9-88bd-754ce8434828 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:10.165584 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 783/1586] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:10 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:10.166686 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1dd171-4cdb-48d9-88bd-754ce8434828 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:10.179438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1dd171-4cdb-48d9-88bd-754ce8434828 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:10.185220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3006a6a-7609-4b00-ac41-09b25fd6c753 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:10.185916 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3006a6a-7609-4b00-ac41-09b25fd6c753 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.186398 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3006a6a-7609-4b00-ac41-09b25fd6c753 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.186741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3006a6a-7609-4b00-ac41-09b25fd6c753 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.187535 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3006a6a-7609-4b00-ac41-09b25fd6c753 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:10.189623 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3006a6a-7609-4b00-ac41-09b25fd6c753 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:10.190807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 804/1587] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:10 2022] POST /identity/v3/projects => generated 379 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:10.192431 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c7bfda-618d-4a8d-b352-a3519b41c3bd admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:10.237248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 784/1588] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:10 2022] POST /identity/v3/projects => generated 377 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:10.239247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3006a6a-7609-4b00-ac41-09b25fd6c753 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:10.275662 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c7bfda-618d-4a8d-b352-a3519b41c3bd admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:10.277664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:10.286379 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:10.320707 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3006a6a-7609-4b00-ac41-09b25fd6c753 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:10.326454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a5e987-4c75-4278-99d4-16b61c9599c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:10.327112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a5e987-4c75-4278-99d4-16b61c9599c0 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.327385 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a5e987-4c75-4278-99d4-16b61c9599c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.327690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a5e987-4c75-4278-99d4-16b61c9599c0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.328301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a5e987-4c75-4278-99d4-16b61c9599c0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:10.330405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a5e987-4c75-4278-99d4-16b61c9599c0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:10.342548 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27460366-f312-43e5-a326-6b454576b666 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:10.343536 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27460366-f312-43e5-a326-6b454576b666 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.343841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27460366-f312-43e5-a326-6b454576b666 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.344117 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27460366-f312-43e5-a326-6b454576b666 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.344821 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27460366-f312-43e5-a326-6b454576b666 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:10.346704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27460366-f312-43e5-a326-6b454576b666 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:10.368089 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 785/1589] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:10 2022] POST /identity/v3/projects => generated 367 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:10.368089 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a5e987-4c75-4278-99d4-16b61c9599c0 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:10.378366 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 805/1590] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:10 2022] POST /identity/v3/projects => generated 381 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:10.380718 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eed14eb-033a-4437-8cfb-b35494232daf None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.381211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eed14eb-033a-4437-8cfb-b35494232daf None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.381717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eed14eb-033a-4437-8cfb-b35494232daf None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.383424 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 806/1591] 10.222.0.85 () {54 vars in 962 bytes} [Tue Aug 30 04:26:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:26:10.384151 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eed14eb-033a-4437-8cfb-b35494232daf None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:10.448245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a5e987-4c75-4278-99d4-16b61c9599c0 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:10.454853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497945eb-041c-48fb-bff2-f0ef4af4d1af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:10.455994 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eed14eb-033a-4437-8cfb-b35494232daf None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:10.457014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497945eb-041c-48fb-bff2-f0ef4af4d1af admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.457859 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497945eb-041c-48fb-bff2-f0ef4af4d1af admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.458206 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497945eb-041c-48fb-bff2-f0ef4af4d1af admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.458974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497945eb-041c-48fb-bff2-f0ef4af4d1af admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:10.461746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497945eb-041c-48fb-bff2-f0ef4af4d1af admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:10.461918 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa535e8-be19-4481-b898-2d0194dcc7da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:10.462633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa535e8-be19-4481-b898-2d0194dcc7da admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.463009 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa535e8-be19-4481-b898-2d0194dcc7da admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.463386 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa535e8-be19-4481-b898-2d0194dcc7da admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.464156 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa535e8-be19-4481-b898-2d0194dcc7da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:10.465994 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa535e8-be19-4481-b898-2d0194dcc7da admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:10.485226 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 807/1592] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:10 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:10.486208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa535e8-be19-4481-b898-2d0194dcc7da admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:10.498038 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 786/1593] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:10 2022] POST /identity/v3/users => generated 286 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:10.498499 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497945eb-041c-48fb-bff2-f0ef4af4d1af admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:10.502517 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa535e8-be19-4481-b898-2d0194dcc7da admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:10.507901 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c462282-c259-401a-95ee-8ac87af5429a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:10.508612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c462282-c259-401a-95ee-8ac87af5429a admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.508997 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c462282-c259-401a-95ee-8ac87af5429a admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.509472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c462282-c259-401a-95ee-8ac87af5429a admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.509723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497945eb-041c-48fb-bff2-f0ef4af4d1af admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:10.510305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c462282-c259-401a-95ee-8ac87af5429a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:10.512376 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c462282-c259-401a-95ee-8ac87af5429a admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:10.514775 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a936560-af34-4865-b021-d8ba6fda14cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:10.515386 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a936560-af34-4865-b021-d8ba6fda14cd admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.515663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a936560-af34-4865-b021-d8ba6fda14cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.515934 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a936560-af34-4865-b021-d8ba6fda14cd admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.516560 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a936560-af34-4865-b021-d8ba6fda14cd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:10.518705 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a936560-af34-4865-b021-d8ba6fda14cd admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:10.541805 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 808/1594] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:10 2022] POST /identity/v3/users => generated 290 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:10.542869 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c462282-c259-401a-95ee-8ac87af5429a admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:10.549964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 787/1595] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:10 2022] POST /identity/v3/users => generated 289 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:10.551173 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a936560-af34-4865-b021-d8ba6fda14cd admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:10.554748 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c462282-c259-401a-95ee-8ac87af5429a admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:10.560424 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7540d26a-1901-4280-bcf3-727058916e6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:10.561197 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7540d26a-1901-4280-bcf3-727058916e6c admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.561582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7540d26a-1901-4280-bcf3-727058916e6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.561974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7540d26a-1901-4280-bcf3-727058916e6c admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.563154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7540d26a-1901-4280-bcf3-727058916e6c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:10.564870 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7540d26a-1901-4280-bcf3-727058916e6c admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:10.576962 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a936560-af34-4865-b021-d8ba6fda14cd admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:10.584467 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a06c34e-af8a-440e-92e8-b7344733c2d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:10.586075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a06c34e-af8a-440e-92e8-b7344733c2d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.586540 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a06c34e-af8a-440e-92e8-b7344733c2d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.586974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a06c34e-af8a-440e-92e8-b7344733c2d8 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.587615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a06c34e-af8a-440e-92e8-b7344733c2d8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:10.590037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a06c34e-af8a-440e-92e8-b7344733c2d8 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:10.593427 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 809/1596] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:10 2022] POST /identity/v3/users => generated 284 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:10.594785 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dd69a8-cb6b-4cf2-8fc9-60da9a165b68 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.595175 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dd69a8-cb6b-4cf2-8fc9-60da9a165b68 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.595548 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dd69a8-cb6b-4cf2-8fc9-60da9a165b68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.626947 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 788/1597] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:10 2022] POST /identity/v3/users => generated 291 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:10.627733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a06c34e-af8a-440e-92e8-b7344733c2d8 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:10.641869 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-16dd69a8-cb6b-4cf2-8fc9-60da9a165b68 None None] MFA Rules not processed for user `4bc4d455ebf041d488b1199ca58358a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:10.643378 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-16dd69a8-cb6b-4cf2-8fc9-60da9a165b68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:10.643964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a06c34e-af8a-440e-92e8-b7344733c2d8 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:10.651495 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3247f6b-7485-4d8b-83d4-e5f9c4fbeb04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:10.652561 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3247f6b-7485-4d8b-83d4-e5f9c4fbeb04 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.653626 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3247f6b-7485-4d8b-83d4-e5f9c4fbeb04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.654076 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3247f6b-7485-4d8b-83d4-e5f9c4fbeb04 admin admin] PATH_INFO: `/v3/projects/27128e3e85bb4712a8f6ee5622576cf5/users/51612cd5460847928b1b533d1b2d3840/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.676911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3247f6b-7485-4d8b-83d4-e5f9c4fbeb04 admin admin] RBAC: Authorizing `identity:create_grant(project_id=27128e3e85bb4712a8f6ee5622576cf5, user_id=51612cd5460847928b1b533d1b2d3840, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:10.681977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3247f6b-7485-4d8b-83d4-e5f9c4fbeb04 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:10.700678 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:10.703876 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 789/1598] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:10 2022] PUT /identity/v3/projects/27128e3e85bb4712a8f6ee5622576cf5/users/51612cd5460847928b1b533d1b2d3840/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:10.706771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3247f6b-7485-4d8b-83d4-e5f9c4fbeb04 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:10.733376 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 810/1599] 10.222.0.85 () {58 vars in 1059 bytes} [Tue Aug 30 04:26:10 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:10.734499 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dd69a8-cb6b-4cf2-8fc9-60da9a165b68 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:10.747001 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dd69a8-cb6b-4cf2-8fc9-60da9a165b68 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:10.771292 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:INSERT=11 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:10.788988 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3247f6b-7485-4d8b-83d4-e5f9c4fbeb04 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:10.795550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dd86e3-d21b-4d8e-9b8c-4e6c7b76020f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:10.796608 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dd86e3-d21b-4d8e-9b8c-4e6c7b76020f admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.797102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dd86e3-d21b-4d8e-9b8c-4e6c7b76020f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.797570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dd86e3-d21b-4d8e-9b8c-4e6c7b76020f admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.798624 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0dd86e3-d21b-4d8e-9b8c-4e6c7b76020f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:10.801373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0dd86e3-d21b-4d8e-9b8c-4e6c7b76020f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:10.812601 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 790/1600] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:10 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:10.816258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dd86e3-d21b-4d8e-9b8c-4e6c7b76020f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:10.827190 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dd86e3-d21b-4d8e-9b8c-4e6c7b76020f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:10.831964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c7d8b7-ddc3-4aa2-823a-981decf7fc94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:10.832591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c7d8b7-ddc3-4aa2-823a-981decf7fc94 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.832859 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c7d8b7-ddc3-4aa2-823a-981decf7fc94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.833178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c7d8b7-ddc3-4aa2-823a-981decf7fc94 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.833843 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c7d8b7-ddc3-4aa2-823a-981decf7fc94 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:10.835425 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c7d8b7-ddc3-4aa2-823a-981decf7fc94 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:10.840339 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b698cf23-cf83-4157-a3bd-fd7f87b481c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:10.842027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b698cf23-cf83-4157-a3bd-fd7f87b481c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.842583 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b698cf23-cf83-4157-a3bd-fd7f87b481c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.843069 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b698cf23-cf83-4157-a3bd-fd7f87b481c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.844414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b698cf23-cf83-4157-a3bd-fd7f87b481c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:10.846824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b698cf23-cf83-4157-a3bd-fd7f87b481c6 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:10.847023 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 791/1601] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:10 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:10.847993 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c7d8b7-ddc3-4aa2-823a-981decf7fc94 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:10.858718 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c7d8b7-ddc3-4aa2-823a-981decf7fc94 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:10.864191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff1e49e-d527-46ac-a23d-82ba112e93d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:10.864798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff1e49e-d527-46ac-a23d-82ba112e93d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.865146 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 811/1602] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:10 2022] GET /identity/v3/roles => generated 1420 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:10.865547 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff1e49e-d527-46ac-a23d-82ba112e93d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.865828 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff1e49e-d527-46ac-a23d-82ba112e93d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.866601 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff1e49e-d527-46ac-a23d-82ba112e93d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:10.867319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b698cf23-cf83-4157-a3bd-fd7f87b481c6 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:10.868213 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff1e49e-d527-46ac-a23d-82ba112e93d9 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:10.881991 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b698cf23-cf83-4157-a3bd-fd7f87b481c6 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:10.882981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 792/1603] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:10 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:10.884268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e6f6d7-a852-4a3e-af70-c954a596e9af None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.885462 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e6f6d7-a852-4a3e-af70-c954a596e9af None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.885896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e6f6d7-a852-4a3e-af70-c954a596e9af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.889717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26151318-9f89-4ffc-a181-f8a3eeffae8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:10.890998 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26151318-9f89-4ffc-a181-f8a3eeffae8e admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:10.892113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26151318-9f89-4ffc-a181-f8a3eeffae8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:10.892775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26151318-9f89-4ffc-a181-f8a3eeffae8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:10.893741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26151318-9f89-4ffc-a181-f8a3eeffae8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:10.897311 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26151318-9f89-4ffc-a181-f8a3eeffae8e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:10.915091 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 812/1604] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:10 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:10.916210 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26151318-9f89-4ffc-a181-f8a3eeffae8e admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:10.922242 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-e9e6f6d7-a852-4a3e-af70-c954a596e9af None None] MFA Rules not processed for user `51612cd5460847928b1b533d1b2d3840`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:10.923641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e9e6f6d7-a852-4a3e-af70-c954a596e9af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:11.010637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 793/1605] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:10 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:11.011826 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e6f6d7-a852-4a3e-af70-c954a596e9af None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:11.021262 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26151318-9f89-4ffc-a181-f8a3eeffae8e admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:11.026002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e6f6d7-a852-4a3e-af70-c954a596e9af None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:11.028816 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43885ec-c015-4e67-8593-549144650a2d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:11.029917 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43885ec-c015-4e67-8593-549144650a2d service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:11.030378 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43885ec-c015-4e67-8593-549144650a2d service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:11.030906 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43885ec-c015-4e67-8593-549144650a2d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:11.034317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681d1fa5-55d6-430a-9d95-9727559c2ccd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:11.034938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681d1fa5-55d6-430a-9d95-9727559c2ccd admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:11.037339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681d1fa5-55d6-430a-9d95-9727559c2ccd admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:11.037941 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681d1fa5-55d6-430a-9d95-9727559c2ccd admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:11.039062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681d1fa5-55d6-430a-9d95-9727559c2ccd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:11.043422 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681d1fa5-55d6-430a-9d95-9727559c2ccd admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:11.064283 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 794/1606] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:11 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:11.064283 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681d1fa5-55d6-430a-9d95-9727559c2ccd admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:11.075990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681d1fa5-55d6-430a-9d95-9727559c2ccd admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:11.081378 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3935b7d0-b188-4e5b-ae7c-434e05081de0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:11.082313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3935b7d0-b188-4e5b-ae7c-434e05081de0 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:11.082702 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3935b7d0-b188-4e5b-ae7c-434e05081de0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:11.083082 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3935b7d0-b188-4e5b-ae7c-434e05081de0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:11.083858 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3935b7d0-b188-4e5b-ae7c-434e05081de0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:11.085830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3935b7d0-b188-4e5b-ae7c-434e05081de0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:11.103277 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 795/1607] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:11 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:11.103277 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3935b7d0-b188-4e5b-ae7c-434e05081de0 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:11.123885 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3935b7d0-b188-4e5b-ae7c-434e05081de0 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:11.132431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8fb210-8a53-4bc8-8549-919b157aefc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:11.133101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8fb210-8a53-4bc8-8549-919b157aefc0 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:11.133400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8fb210-8a53-4bc8-8549-919b157aefc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:11.133675 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8fb210-8a53-4bc8-8549-919b157aefc0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:11.134388 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8fb210-8a53-4bc8-8549-919b157aefc0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:11.134740 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43885ec-c015-4e67-8593-549144650a2d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:11.136637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8fb210-8a53-4bc8-8549-919b157aefc0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:11.137761 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43885ec-c015-4e67-8593-549144650a2d service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:11.148748 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:11.155375 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 796/1608] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:11 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:11.161889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8fb210-8a53-4bc8-8549-919b157aefc0 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:11.168017 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 813/1609] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:10 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 252 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:11.169560 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43885ec-c015-4e67-8593-549144650a2d service nova] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:11.183076 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43885ec-c015-4e67-8593-549144650a2d service nova] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:11.188777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87092b94-541f-417d-a0cf-12a135c8da7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:11.189565 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87092b94-541f-417d-a0cf-12a135c8da7a admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:11.189957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87092b94-541f-417d-a0cf-12a135c8da7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:11.190340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87092b94-541f-417d-a0cf-12a135c8da7a admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:11.191094 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87092b94-541f-417d-a0cf-12a135c8da7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:11.193123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87092b94-541f-417d-a0cf-12a135c8da7a admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:11.195333 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8fb210-8a53-4bc8-8549-919b157aefc0 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:11.200816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b433ef5c-fd71-4ec4-a73d-5bcc3cffbaf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:11.201512 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b433ef5c-fd71-4ec4-a73d-5bcc3cffbaf9 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:11.201848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b433ef5c-fd71-4ec4-a73d-5bcc3cffbaf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:11.202129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b433ef5c-fd71-4ec4-a73d-5bcc3cffbaf9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:11.203245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b433ef5c-fd71-4ec4-a73d-5bcc3cffbaf9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:11.205746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b433ef5c-fd71-4ec4-a73d-5bcc3cffbaf9 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:11.206308 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 814/1610] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:11 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:11.207436 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87092b94-541f-417d-a0cf-12a135c8da7a admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:11.218006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87092b94-541f-417d-a0cf-12a135c8da7a admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:11.224310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 797/1611] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:11 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:11.225337 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b433ef5c-fd71-4ec4-a73d-5bcc3cffbaf9 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:11.231233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb12606-e50d-49ab-945f-87d8eddfe14c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:11.232984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb12606-e50d-49ab-945f-87d8eddfe14c admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:11.233704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb12606-e50d-49ab-945f-87d8eddfe14c admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:11.234292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb12606-e50d-49ab-945f-87d8eddfe14c admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:11.235159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb12606-e50d-49ab-945f-87d8eddfe14c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:11.241311 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b433ef5c-fd71-4ec4-a73d-5bcc3cffbaf9 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:11.242673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb12606-e50d-49ab-945f-87d8eddfe14c admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:11.247999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6283e6-205e-4d06-ae0e-9593afdc7742 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:11.249386 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6283e6-205e-4d06-ae0e-9593afdc7742 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:11.249673 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6283e6-205e-4d06-ae0e-9593afdc7742 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:11.249950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6283e6-205e-4d06-ae0e-9593afdc7742 admin admin] PATH_INFO: `/v3/projects/e35dc7910f7448f1b0646bb291261be0/users/e7284a7792a04e9493c54eae58ca89a6/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:11.296842 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6283e6-205e-4d06-ae0e-9593afdc7742 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e35dc7910f7448f1b0646bb291261be0, user_id=e7284a7792a04e9493c54eae58ca89a6, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:11.304138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6283e6-205e-4d06-ae0e-9593afdc7742 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:11.328338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 798/1612] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:11 2022] PUT /identity/v3/projects/e35dc7910f7448f1b0646bb291261be0/users/e7284a7792a04e9493c54eae58ca89a6/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:11.338455 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6283e6-205e-4d06-ae0e-9593afdc7742 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:11.342016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 815/1613] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:11 2022] POST /identity/v3/projects => generated 405 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:11.342960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb12606-e50d-49ab-945f-87d8eddfe14c admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:11.430260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:11.463352 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb12606-e50d-49ab-945f-87d8eddfe14c admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:11.466133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6283e6-205e-4d06-ae0e-9593afdc7742 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:11.470424 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4200669-d5be-40d5-8eee-d42b52e358fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:11.471109 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4200669-d5be-40d5-8eee-d42b52e358fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:11.471407 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4200669-d5be-40d5-8eee-d42b52e358fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:11.471733 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4200669-d5be-40d5-8eee-d42b52e358fc admin admin] PATH_INFO: `/v3/projects/959853ec613f47819ed2883c8539b24e/users/298b05a8580946f4ab2ed66f0b9a3c2b/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:11.472190 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbbe1e4-2f2c-455c-97f4-2bc3ede9979c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:11.473896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbbe1e4-2f2c-455c-97f4-2bc3ede9979c admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:11.474277 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbbe1e4-2f2c-455c-97f4-2bc3ede9979c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:11.474889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbbe1e4-2f2c-455c-97f4-2bc3ede9979c admin admin] PATH_INFO: `/v3/projects/d7deffddf9934a028bfdb0b1c965c618/users/8ef7a83a87d44fc0a5647cb60057c036/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:11.492906 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4200669-d5be-40d5-8eee-d42b52e358fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=959853ec613f47819ed2883c8539b24e, user_id=298b05a8580946f4ab2ed66f0b9a3c2b, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:11.496441 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbbe1e4-2f2c-455c-97f4-2bc3ede9979c admin admin] RBAC: Authorizing `identity:create_grant(project_id=d7deffddf9934a028bfdb0b1c965c618, user_id=8ef7a83a87d44fc0a5647cb60057c036, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:11.496617 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4200669-d5be-40d5-8eee-d42b52e358fc admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:11.499419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbbe1e4-2f2c-455c-97f4-2bc3ede9979c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:11.522502 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 799/1614] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:11 2022] PUT /identity/v3/projects/d7deffddf9934a028bfdb0b1c965c618/users/8ef7a83a87d44fc0a5647cb60057c036/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:11.523572 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbbe1e4-2f2c-455c-97f4-2bc3ede9979c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:11.526068 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 816/1615] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:11 2022] PUT /identity/v3/projects/959853ec613f47819ed2883c8539b24e/users/298b05a8580946f4ab2ed66f0b9a3c2b/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:11.527150 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4200669-d5be-40d5-8eee-d42b52e358fc admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:11.683050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbbe1e4-2f2c-455c-97f4-2bc3ede9979c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:11.683050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc03a0e-42cd-4c81-931a-497796157191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:11.683050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc03a0e-42cd-4c81-931a-497796157191 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:11.683050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc03a0e-42cd-4c81-931a-497796157191 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:11.683818 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc03a0e-42cd-4c81-931a-497796157191 admin admin] PATH_INFO: `/v3/projects/d1de048add3b4fc79401df1d0952f67a/users/aafd1566d1ed4174abe8de556bf8d001/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:11.684022 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4200669-d5be-40d5-8eee-d42b52e358fc admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:11.684022 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadc675a-d89e-40cc-b34d-43d4b48c4a66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:11.684022 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadc675a-d89e-40cc-b34d-43d4b48c4a66 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:11.684022 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadc675a-d89e-40cc-b34d-43d4b48c4a66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:11.684589 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadc675a-d89e-40cc-b34d-43d4b48c4a66 admin admin] PATH_INFO: `/v3/projects/44cec317ac97406d8909fc153756808b/users/d8444f6398e04554a7907a2d3e0eb1cf/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:11.690495 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc03a0e-42cd-4c81-931a-497796157191 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1de048add3b4fc79401df1d0952f67a, user_id=aafd1566d1ed4174abe8de556bf8d001, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:11.694757 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc03a0e-42cd-4c81-931a-497796157191 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:11.715182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadc675a-d89e-40cc-b34d-43d4b48c4a66 admin admin] RBAC: Authorizing `identity:create_grant(project_id=44cec317ac97406d8909fc153756808b, user_id=d8444f6398e04554a7907a2d3e0eb1cf, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:11.715374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 800/1616] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:11 2022] PUT /identity/v3/projects/d1de048add3b4fc79401df1d0952f67a/users/aafd1566d1ed4174abe8de556bf8d001/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:11.716318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc03a0e-42cd-4c81-931a-497796157191 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:11.728301 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadc675a-d89e-40cc-b34d-43d4b48c4a66 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:11.761564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 817/1617] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:11 2022] PUT /identity/v3/projects/44cec317ac97406d8909fc153756808b/users/d8444f6398e04554a7907a2d3e0eb1cf/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:11.763463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77af3c3d-6c96-4e59-9604-dabd38d0e566 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:11.763826 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77af3c3d-6c96-4e59-9604-dabd38d0e566 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:11.764211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77af3c3d-6c96-4e59-9604-dabd38d0e566 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:11.817206 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-77af3c3d-6c96-4e59-9604-dabd38d0e566 None None] MFA Rules not processed for user `e7284a7792a04e9493c54eae58ca89a6`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:11.818924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-77af3c3d-6c96-4e59-9604-dabd38d0e566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:11.819052 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc03a0e-42cd-4c81-931a-497796157191 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:11.857338 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:11.881067 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:11.891904 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b65d82-3422-400f-ae6c-1a2294abb61c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:11.893421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b65d82-3422-400f-ae6c-1a2294abb61c admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:11.894557 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b65d82-3422-400f-ae6c-1a2294abb61c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:11.894974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b65d82-3422-400f-ae6c-1a2294abb61c admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:11.895749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b65d82-3422-400f-ae6c-1a2294abb61c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:11.898705 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b65d82-3422-400f-ae6c-1a2294abb61c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:11.928108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 801/1618] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:11 2022] POST /identity/v3/users => generated 309 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:11.929533 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2328f2-ed13-4de2-9ef5-d0605c0a935c None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:11.929823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2328f2-ed13-4de2-9ef5-d0605c0a935c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:11.930097 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2328f2-ed13-4de2-9ef5-d0605c0a935c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:11.949157 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 818/1619] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:11 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:11.950541 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d675047-1122-43f4-ae47-ca5277f172f5 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:11.950831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d675047-1122-43f4-ae47-ca5277f172f5 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:11.951110 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d675047-1122-43f4-ae47-ca5277f172f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:11.971011 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5a2328f2-ed13-4de2-9ef5-d0605c0a935c None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:11.974565 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5a2328f2-ed13-4de2-9ef5-d0605c0a935c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:11.987294 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 802/1620] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:11.987294 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7752de85-44cb-423a-ad49-907efd475a08 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:11.987294 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7752de85-44cb-423a-ad49-907efd475a08 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:11.987294 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7752de85-44cb-423a-ad49-907efd475a08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:12.002948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-5d675047-1122-43f4-ae47-ca5277f172f5 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:12.004372 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-5d675047-1122-43f4-ae47-ca5277f172f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:12.015097 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 819/1621] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:12.016608 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a1a6f2-ed87-4f33-822b-5806b8dd6e61 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:12.016919 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a1a6f2-ed87-4f33-822b-5806b8dd6e61 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:12.017206 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a1a6f2-ed87-4f33-822b-5806b8dd6e61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:12.024671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-7752de85-44cb-423a-ad49-907efd475a08 None None] MFA Rules not processed for user `aafd1566d1ed4174abe8de556bf8d001`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:12.026171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7752de85-44cb-423a-ad49-907efd475a08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:12.064517 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-02a1a6f2-ed87-4f33-822b-5806b8dd6e61 None None] MFA Rules not processed for user `d8444f6398e04554a7907a2d3e0eb1cf`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:12.066005 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-02a1a6f2-ed87-4f33-822b-5806b8dd6e61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:12.128523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 803/1622] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:11 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:12.130849 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1aec10-10d0-4aad-b0b7-eb734a0b4471 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:12.131136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1aec10-10d0-4aad-b0b7-eb734a0b4471 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:12.131412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1aec10-10d0-4aad-b0b7-eb734a0b4471 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:12.133158 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 804/1623] 10.222.0.85 () {54 vars in 962 bytes} [Tue Aug 30 04:26:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:26:12.133915 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1aec10-10d0-4aad-b0b7-eb734a0b4471 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:12.150056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1aec10-10d0-4aad-b0b7-eb734a0b4471 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:12.155679 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa8dcd3-487b-4aa0-b936-5700ca9f6349 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:12.156341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa8dcd3-487b-4aa0-b936-5700ca9f6349 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:12.156657 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa8dcd3-487b-4aa0-b936-5700ca9f6349 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:12.156934 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa8dcd3-487b-4aa0-b936-5700ca9f6349 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:12.158024 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa8dcd3-487b-4aa0-b936-5700ca9f6349 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:12.159529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa8dcd3-487b-4aa0-b936-5700ca9f6349 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:12.173862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 820/1624] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:12 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:12.175401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a1a6f2-ed87-4f33-822b-5806b8dd6e61 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:12.179652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 805/1625] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:12 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:12.184482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa8dcd3-487b-4aa0-b936-5700ca9f6349 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:12.187070 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6fa8dcd3-487b-4aa0-b936-5700ca9f6349 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:12.284996 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a1a6f2-ed87-4f33-822b-5806b8dd6e61 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:12.286082 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa8dcd3-487b-4aa0-b936-5700ca9f6349 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:12.292699 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbd4be1-25fb-4105-b715-bcb2e8ab11f0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:12.293685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbd4be1-25fb-4105-b715-bcb2e8ab11f0 service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:12.294136 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbd4be1-25fb-4105-b715-bcb2e8ab11f0 service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:12.294669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbd4be1-25fb-4105-b715-bcb2e8ab11f0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:12.296778 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d248f4b5-9181-413c-8258-2353d4be1ac0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:12.297509 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d248f4b5-9181-413c-8258-2353d4be1ac0 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:12.297887 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d248f4b5-9181-413c-8258-2353d4be1ac0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:12.298255 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d248f4b5-9181-413c-8258-2353d4be1ac0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:12.309756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbd4be1-25fb-4105-b715-bcb2e8ab11f0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:12.311593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbd4be1-25fb-4105-b715-bcb2e8ab11f0 service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:12.311816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d248f4b5-9181-413c-8258-2353d4be1ac0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:12.313607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d248f4b5-9181-413c-8258-2353d4be1ac0 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:12.322889 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 821/1626] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:12 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:12.323766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbd4be1-25fb-4105-b715-bcb2e8ab11f0 service nova] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:12.327268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 806/1627] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:12.328260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d248f4b5-9181-413c-8258-2353d4be1ac0 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:12.334388 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbd4be1-25fb-4105-b715-bcb2e8ab11f0 service nova] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:12.338747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf953f06-a958-46bc-aa17-c8934151d5c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:12.339502 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf953f06-a958-46bc-aa17-c8934151d5c7 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:12.339782 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf953f06-a958-46bc-aa17-c8934151d5c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:12.340054 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf953f06-a958-46bc-aa17-c8934151d5c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:12.343711 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d248f4b5-9181-413c-8258-2353d4be1ac0 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:12.349476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56a0e57-74f2-4dae-ba7a-c25c7ee03811 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:12.350916 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56a0e57-74f2-4dae-ba7a-c25c7ee03811 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:12.351281 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56a0e57-74f2-4dae-ba7a-c25c7ee03811 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:12.351658 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56a0e57-74f2-4dae-ba7a-c25c7ee03811 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:12.352403 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56a0e57-74f2-4dae-ba7a-c25c7ee03811 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:12.354770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56a0e57-74f2-4dae-ba7a-c25c7ee03811 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:12.355940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf953f06-a958-46bc-aa17-c8934151d5c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:12.359605 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf953f06-a958-46bc-aa17-c8934151d5c7 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:12.395052 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 822/1628] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:12.396627 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1194f89-7089-49f3-9632-54a0d1aebd94 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:12.396904 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1194f89-7089-49f3-9632-54a0d1aebd94 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:12.397207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1194f89-7089-49f3-9632-54a0d1aebd94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:12.397347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 807/1629] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:12 2022] POST /identity/v3/projects => generated 381 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:12.398566 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56a0e57-74f2-4dae-ba7a-c25c7ee03811 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:12.430875 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-c1194f89-7089-49f3-9632-54a0d1aebd94 None None] MFA Rules not processed for user `4bc4d455ebf041d488b1199ca58358a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:12.432522 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c1194f89-7089-49f3-9632-54a0d1aebd94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:12.457987 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:12.460194 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:12.489964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56a0e57-74f2-4dae-ba7a-c25c7ee03811 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:12.503853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ee2430-245e-46cb-a773-6590f1f5af56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:12.505913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ee2430-245e-46cb-a773-6590f1f5af56 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:12.506363 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ee2430-245e-46cb-a773-6590f1f5af56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:12.506805 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ee2430-245e-46cb-a773-6590f1f5af56 admin admin] PATH_INFO: `/v3/projects/21c540078fc04f65b8e365c978b333cd/users/3cbbf4426a06411caa8f453d4a27590a/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:12.532006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ee2430-245e-46cb-a773-6590f1f5af56 admin admin] RBAC: Authorizing `identity:create_grant(project_id=21c540078fc04f65b8e365c978b333cd, user_id=3cbbf4426a06411caa8f453d4a27590a, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:12.537855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ee2430-245e-46cb-a773-6590f1f5af56 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:12.557631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 808/1630] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:12 2022] PUT /identity/v3/projects/21c540078fc04f65b8e365c978b333cd/users/3cbbf4426a06411caa8f453d4a27590a/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:12.559544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ee2430-245e-46cb-a773-6590f1f5af56 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:12.584953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 823/1631] 10.222.0.85 () {58 vars in 1059 bytes} [Tue Aug 30 04:26:12 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:12.585933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1194f89-7089-49f3-9632-54a0d1aebd94 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:12.598171 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1194f89-7089-49f3-9632-54a0d1aebd94 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:12.641402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ee2430-245e-46cb-a773-6590f1f5af56 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:12.646853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d07264-68c7-4ed2-9908-e4bdb6b8d96d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:12.647813 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d07264-68c7-4ed2-9908-e4bdb6b8d96d service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:12.648346 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d07264-68c7-4ed2-9908-e4bdb6b8d96d service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:12.648863 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d07264-68c7-4ed2-9908-e4bdb6b8d96d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:12.743688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f36373-ca5c-4a08-9d84-6e06aabb4886 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:12.746994 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f36373-ca5c-4a08-9d84-6e06aabb4886 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:12.747565 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f36373-ca5c-4a08-9d84-6e06aabb4886 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:12.747990 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f36373-ca5c-4a08-9d84-6e06aabb4886 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:12.748809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f36373-ca5c-4a08-9d84-6e06aabb4886 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:12.749855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d07264-68c7-4ed2-9908-e4bdb6b8d96d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:12.751152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f36373-ca5c-4a08-9d84-6e06aabb4886 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:12.753113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d07264-68c7-4ed2-9908-e4bdb6b8d96d service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:12.785180 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 809/1632] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:12 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 226 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:12.786574 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee509de-1c38-4128-9a5b-526fd6e4d1b9 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:12.786937 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee509de-1c38-4128-9a5b-526fd6e4d1b9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:12.787217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee509de-1c38-4128-9a5b-526fd6e4d1b9 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:12.788796 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 810/1633] 10.222.0.85 () {54 vars in 912 bytes} [Tue Aug 30 04:26:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:26:12.790151 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee509de-1c38-4128-9a5b-526fd6e4d1b9 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:12.801941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 824/1634] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:12 2022] POST /identity/v3/users => generated 297 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:12.804438 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4fc12d-9fa0-4476-9e8e-b9eab170fb75 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:12.805611 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4fc12d-9fa0-4476-9e8e-b9eab170fb75 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:12.805904 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4fc12d-9fa0-4476-9e8e-b9eab170fb75 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:12.807488 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 825/1635] 10.222.0.85 () {54 vars in 912 bytes} [Tue Aug 30 04:26:12 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:26:12.817577 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec773025-a0d5-4ebc-bd94-724d07bcf309 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:12.818200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec773025-a0d5-4ebc-bd94-724d07bcf309 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:12.818200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec773025-a0d5-4ebc-bd94-724d07bcf309 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:12.820227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee509de-1c38-4128-9a5b-526fd6e4d1b9 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:12.833685 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7d7759-4bdf-412d-bcf5-94610b52430d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:12.834310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7d7759-4bdf-412d-bcf5-94610b52430d admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:12.834588 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7d7759-4bdf-412d-bcf5-94610b52430d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:12.834856 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7d7759-4bdf-412d-bcf5-94610b52430d admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:12.835501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7d7759-4bdf-412d-bcf5-94610b52430d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:12.837122 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7d7759-4bdf-412d-bcf5-94610b52430d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:12.851190 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-ec773025-a0d5-4ebc-bd94-724d07bcf309 None None] MFA Rules not processed for user `4bc4d455ebf041d488b1199ca58358a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:12.852732 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ec773025-a0d5-4ebc-bd94-724d07bcf309 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:12.856826 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 811/1636] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:12 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:12.857817 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7d7759-4bdf-412d-bcf5-94610b52430d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:12.883931 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7d7759-4bdf-412d-bcf5-94610b52430d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:12.889253 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17948184-a2d5-47fd-b37f-3d5d8eada30f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:12.889902 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17948184-a2d5-47fd-b37f-3d5d8eada30f admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:12.890246 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17948184-a2d5-47fd-b37f-3d5d8eada30f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:12.890551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17948184-a2d5-47fd-b37f-3d5d8eada30f admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:12.891339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17948184-a2d5-47fd-b37f-3d5d8eada30f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:12.893178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17948184-a2d5-47fd-b37f-3d5d8eada30f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:12.909317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 812/1637] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:12 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:12.910912 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3037211-b2c1-4a52-841e-f961ffc40cd5 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:12.911193 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3037211-b2c1-4a52-841e-f961ffc40cd5 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:12.911532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3037211-b2c1-4a52-841e-f961ffc40cd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:12.945356 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-e3037211-b2c1-4a52-841e-f961ffc40cd5 None None] MFA Rules not processed for user `4bc4d455ebf041d488b1199ca58358a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:12.951032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e3037211-b2c1-4a52-841e-f961ffc40cd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:13.023266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 826/1638] 10.222.0.85 () {58 vars in 1009 bytes} [Tue Aug 30 04:26:12 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:13.025759 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec773025-a0d5-4ebc-bd94-724d07bcf309 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:13.047903 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec773025-a0d5-4ebc-bd94-724d07bcf309 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:13.048574 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:13.060748 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fae581-e934-423a-ab13-4a33fc31258a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:13.063040 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fae581-e934-423a-ab13-4a33fc31258a admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:13.063318 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fae581-e934-423a-ab13-4a33fc31258a admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:13.064405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fae581-e934-423a-ab13-4a33fc31258a admin admin] PATH_INFO: `/v3/projects/21c540078fc04f65b8e365c978b333cd/users/3cbbf4426a06411caa8f453d4a27590a/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:13.071417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fae581-e934-423a-ab13-4a33fc31258a admin admin] RBAC: Authorizing `identity:create_grant(project_id=21c540078fc04f65b8e365c978b333cd, user_id=3cbbf4426a06411caa8f453d4a27590a, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:13.077681 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fae581-e934-423a-ab13-4a33fc31258a admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:13.101472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 813/1639] 10.222.0.85 () {58 vars in 1009 bytes} [Tue Aug 30 04:26:12 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:13.102894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3037211-b2c1-4a52-841e-f961ffc40cd5 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:13.113868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 827/1640] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:13 2022] PUT /identity/v3/projects/21c540078fc04f65b8e365c978b333cd/users/3cbbf4426a06411caa8f453d4a27590a/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:13.114923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fae581-e934-423a-ab13-4a33fc31258a admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:13.142665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3037211-b2c1-4a52-841e-f961ffc40cd5 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:13.173184 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:13.265106 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fae581-e934-423a-ab13-4a33fc31258a admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:13.284096 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b66e0b-a90c-4890-98e5-9da124794074 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:13.284621 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b66e0b-a90c-4890-98e5-9da124794074 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:13.285738 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b66e0b-a90c-4890-98e5-9da124794074 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:13.286072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b66e0b-a90c-4890-98e5-9da124794074 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:13.297611 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea36b8bb-023a-4725-83ff-2eea91c7ae89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:13.298308 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea36b8bb-023a-4725-83ff-2eea91c7ae89 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:13.298738 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea36b8bb-023a-4725-83ff-2eea91c7ae89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:13.298873 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea36b8bb-023a-4725-83ff-2eea91c7ae89 admin admin] PATH_INFO: `/v3/projects/9d68adf14afe47df9c512149e4d01002/users/f806c5ab9e804959a93e38eb1c1e0159/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:13.350121 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea36b8bb-023a-4725-83ff-2eea91c7ae89 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9d68adf14afe47df9c512149e4d01002, user_id=f806c5ab9e804959a93e38eb1c1e0159, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:13.352875 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea36b8bb-023a-4725-83ff-2eea91c7ae89 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:13.398630 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 814/1641] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:13 2022] PUT /identity/v3/projects/9d68adf14afe47df9c512149e4d01002/users/f806c5ab9e804959a93e38eb1c1e0159/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:13.399141 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679b4b03-cf43-40ab-989f-0b43a1938ba1 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:13.399275 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679b4b03-cf43-40ab-989f-0b43a1938ba1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:13.399464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679b4b03-cf43-40ab-989f-0b43a1938ba1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:13.476727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-679b4b03-cf43-40ab-989f-0b43a1938ba1 None None] MFA Rules not processed for user `3cbbf4426a06411caa8f453d4a27590a`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:13.480191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-679b4b03-cf43-40ab-989f-0b43a1938ba1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:13.498127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b66e0b-a90c-4890-98e5-9da124794074 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:13.502488 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b66e0b-a90c-4890-98e5-9da124794074 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:13.544661 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 828/1642] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:13.550604 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b66e0b-a90c-4890-98e5-9da124794074 service cinder] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:13.591156 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b66e0b-a90c-4890-98e5-9da124794074 service cinder] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:13.706646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 815/1643] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:13 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 310 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:13.751705 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029dbb80-64fe-4bf9-802e-f78153f9f578 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:13.755930 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029dbb80-64fe-4bf9-802e-f78153f9f578 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:13.755930 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029dbb80-64fe-4bf9-802e-f78153f9f578 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:13.756160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029dbb80-64fe-4bf9-802e-f78153f9f578 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:13.758001 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029dbb80-64fe-4bf9-802e-f78153f9f578 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:13.761433 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029dbb80-64fe-4bf9-802e-f78153f9f578 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:13.789547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 829/1644] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:13 2022] GET /identity/v3/roles => generated 1420 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:13.793785 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679b4b03-cf43-40ab-989f-0b43a1938ba1 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:13.810617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679b4b03-cf43-40ab-989f-0b43a1938ba1 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:13.819233 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4709258f-8c77-45cd-8ddc-ef022f601bc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:13.819865 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4709258f-8c77-45cd-8ddc-ef022f601bc0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:13.820835 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4709258f-8c77-45cd-8ddc-ef022f601bc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:13.821524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4709258f-8c77-45cd-8ddc-ef022f601bc0 admin admin] PATH_INFO: `/v3/projects/9d68adf14afe47df9c512149e4d01002/users/f806c5ab9e804959a93e38eb1c1e0159/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:13.826523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4709258f-8c77-45cd-8ddc-ef022f601bc0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9d68adf14afe47df9c512149e4d01002, user_id=f806c5ab9e804959a93e38eb1c1e0159, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:13.843455 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4709258f-8c77-45cd-8ddc-ef022f601bc0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:13.859615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029dbb80-64fe-4bf9-802e-f78153f9f578 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:13.873081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 816/1645] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:13 2022] PUT /identity/v3/projects/9d68adf14afe47df9c512149e4d01002/users/f806c5ab9e804959a93e38eb1c1e0159/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:13.881511 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e86010-71a3-4d22-8764-55a01dc49ea8 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:13.882101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e86010-71a3-4d22-8764-55a01dc49ea8 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:13.894773 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e86010-71a3-4d22-8764-55a01dc49ea8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:13.968620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-98e86010-71a3-4d22-8764-55a01dc49ea8 None None] MFA Rules not processed for user `f806c5ab9e804959a93e38eb1c1e0159`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:13.974383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-98e86010-71a3-4d22-8764-55a01dc49ea8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:14.060904 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029dbb80-64fe-4bf9-802e-f78153f9f578 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:14.062403 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:14.086713 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:14.142523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 817/1646] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:13 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 262 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:14.193375 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38137146-35c7-4109-a665-762edb2b2566 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:14.194562 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38137146-35c7-4109-a665-762edb2b2566 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:14.195367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38137146-35c7-4109-a665-762edb2b2566 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:14.196172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38137146-35c7-4109-a665-762edb2b2566 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:14.293032 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38137146-35c7-4109-a665-762edb2b2566 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:14.295793 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38137146-35c7-4109-a665-762edb2b2566 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:14.321519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 830/1647] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 462 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:26:14.396322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e86010-71a3-4d22-8764-55a01dc49ea8 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:14.415372 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e86010-71a3-4d22-8764-55a01dc49ea8 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:14.420047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8f3753-b629-4f39-9417-4de7ed07a5d5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:14.421096 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8f3753-b629-4f39-9417-4de7ed07a5d5 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:14.426775 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8f3753-b629-4f39-9417-4de7ed07a5d5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:14.427370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8f3753-b629-4f39-9417-4de7ed07a5d5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:14.511476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8f3753-b629-4f39-9417-4de7ed07a5d5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:14.513345 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8f3753-b629-4f39-9417-4de7ed07a5d5 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:14.553374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 818/1648] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:14 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:14.662276 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38137146-35c7-4109-a665-762edb2b2566 service cinder] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:14.676405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38137146-35c7-4109-a665-762edb2b2566 service cinder] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:14.685642 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ab4279-4a86-4e3c-8e90-f71d47105396 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:14.686400 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ab4279-4a86-4e3c-8e90-f71d47105396 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:14.686704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ab4279-4a86-4e3c-8e90-f71d47105396 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:14.688465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ab4279-4a86-4e3c-8e90-f71d47105396 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:14.717090 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ab4279-4a86-4e3c-8e90-f71d47105396 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:14.720643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ab4279-4a86-4e3c-8e90-f71d47105396 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:14.754070 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 831/1649] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:14 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:15.024427 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8f3753-b629-4f39-9417-4de7ed07a5d5 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:15.109415 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ab4279-4a86-4e3c-8e90-f71d47105396 service cinder] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:15.119663 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ab4279-4a86-4e3c-8e90-f71d47105396 service cinder] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:15.135892 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4e085a-d377-482e-931c-9be7f4780a2e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:15.136521 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e085a-d377-482e-931c-9be7f4780a2e service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:15.136798 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e085a-d377-482e-931c-9be7f4780a2e service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:15.137082 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e085a-d377-482e-931c-9be7f4780a2e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:15.199758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8f3753-b629-4f39-9417-4de7ed07a5d5 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:15.205644 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766af8f3-dbc0-4f2c-9b2e-057b19c9f2f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:15.206707 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766af8f3-dbc0-4f2c-9b2e-057b19c9f2f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:15.207684 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766af8f3-dbc0-4f2c-9b2e-057b19c9f2f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:15.208120 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766af8f3-dbc0-4f2c-9b2e-057b19c9f2f0 admin admin] PATH_INFO: `/v3/users/37df071b935f4a51b81e5130d9c44e1f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:15.209676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766af8f3-dbc0-4f2c-9b2e-057b19c9f2f0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=37df071b935f4a51b81e5130d9c44e1f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:15.211046 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: WARNING py.warnings [None req-766af8f3-dbc0-4f2c-9b2e-057b19c9f2f0 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 04:26:15.211046 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: warnings.warn(msg) Aug 30 04:26:15.211046 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: Aug 30 04:26:15.212571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766af8f3-dbc0-4f2c-9b2e-057b19c9f2f0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:15.302106 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-766af8f3-dbc0-4f2c-9b2e-057b19c9f2f0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '37df071b935f4a51b81e5130d9c44e1f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:15.319577 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4e085a-d377-482e-931c-9be7f4780a2e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:15.321370 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4e085a-d377-482e-931c-9be7f4780a2e service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:15.329327 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:DELETE=8 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:15.333022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-766af8f3-dbc0-4f2c-9b2e-057b19c9f2f0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '37df071b935f4a51b81e5130d9c44e1f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:15.347029 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 832/1650] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:15 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:26:15.348624 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4e085a-d377-482e-931c-9be7f4780a2e service cinder] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:15.349016 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-766af8f3-dbc0-4f2c-9b2e-057b19c9f2f0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '37df071b935f4a51b81e5130d9c44e1f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:15.360566 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-766af8f3-dbc0-4f2c-9b2e-057b19c9f2f0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '37df071b935f4a51b81e5130d9c44e1f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:15.362029 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-766af8f3-dbc0-4f2c-9b2e-057b19c9f2f0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '37df071b935f4a51b81e5130d9c44e1f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:15.377070 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4e085a-d377-482e-931c-9be7f4780a2e service cinder] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:15.377695 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 819/1651] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:15 2022] DELETE /identity/v3/users/37df071b935f4a51b81e5130d9c44e1f => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:15.379584 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcfb3ae-ce93-4d7a-8b42-d8b0989a8b02 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:15.380068 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcfb3ae-ce93-4d7a-8b42-d8b0989a8b02 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:15.380706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcfb3ae-ce93-4d7a-8b42-d8b0989a8b02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:15.412523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-0bcfb3ae-ce93-4d7a-8b42-d8b0989a8b02 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:15.413950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0bcfb3ae-ce93-4d7a-8b42-d8b0989a8b02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:15.466406 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:15.527875 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4797c1b-464c-4d9d-bbbb-10e5aa477f73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:15.531626 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4797c1b-464c-4d9d-bbbb-10e5aa477f73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:15.535050 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4797c1b-464c-4d9d-bbbb-10e5aa477f73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:15.535354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4797c1b-464c-4d9d-bbbb-10e5aa477f73 admin admin] PATH_INFO: `/v3/users/51612cd5460847928b1b533d1b2d3840` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:15.540042 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4797c1b-464c-4d9d-bbbb-10e5aa477f73 admin admin] RBAC: Authorizing `identity:delete_user(user_id=51612cd5460847928b1b533d1b2d3840)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:15.545859 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4797c1b-464c-4d9d-bbbb-10e5aa477f73 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:15.599771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 820/1652] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:15.619219 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcfb3ae-ce93-4d7a-8b42-d8b0989a8b02 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:15.624440 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0bcfb3ae-ce93-4d7a-8b42-d8b0989a8b02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:15.666193 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d4797c1b-464c-4d9d-bbbb-10e5aa477f73 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '51612cd5460847928b1b533d1b2d3840'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:15.688816 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d4797c1b-464c-4d9d-bbbb-10e5aa477f73 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '51612cd5460847928b1b533d1b2d3840'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:15.689592 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d4797c1b-464c-4d9d-bbbb-10e5aa477f73 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '51612cd5460847928b1b533d1b2d3840'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:15.733840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d4797c1b-464c-4d9d-bbbb-10e5aa477f73 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '51612cd5460847928b1b533d1b2d3840'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:15.736945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d4797c1b-464c-4d9d-bbbb-10e5aa477f73 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '51612cd5460847928b1b533d1b2d3840'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:15.760637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 833/1653] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:15 2022] DELETE /identity/v3/users/51612cd5460847928b1b533d1b2d3840 => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:15.762043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64e5c0c-dda8-4cf6-a691-9a1e41dbcd25 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:15.762332 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64e5c0c-dda8-4cf6-a691-9a1e41dbcd25 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:15.762765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64e5c0c-dda8-4cf6-a691-9a1e41dbcd25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:15.792540 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-c64e5c0c-dda8-4cf6-a691-9a1e41dbcd25 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:15.793915 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c64e5c0c-dda8-4cf6-a691-9a1e41dbcd25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:15.833807 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcfb3ae-ce93-4d7a-8b42-d8b0989a8b02 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:15.877007 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 834/1654] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:15.900336 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a6bcb4-aa52-442d-a346-d9c9d662212e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:15.901094 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a6bcb4-aa52-442d-a346-d9c9d662212e service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:15.901566 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a6bcb4-aa52-442d-a346-d9c9d662212e service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:15.902287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a6bcb4-aa52-442d-a346-d9c9d662212e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:15.904138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b6a6bcb4-aa52-442d-a346-d9c9d662212e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:15.907172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64e5c0c-dda8-4cf6-a691-9a1e41dbcd25 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:15.908659 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c64e5c0c-dda8-4cf6-a691-9a1e41dbcd25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:15.920489 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a6bcb4-aa52-442d-a346-d9c9d662212e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:15.923260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a6bcb4-aa52-442d-a346-d9c9d662212e service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:15.929788 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64e5c0c-dda8-4cf6-a691-9a1e41dbcd25 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:15.944471 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 821/1655] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:15.952157 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174cf110-e138-4189-807d-b7f3d5eb1554 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:15.952502 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174cf110-e138-4189-807d-b7f3d5eb1554 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:15.952785 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174cf110-e138-4189-807d-b7f3d5eb1554 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:15.953098 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174cf110-e138-4189-807d-b7f3d5eb1554 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:15.971172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174cf110-e138-4189-807d-b7f3d5eb1554 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:15.975967 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174cf110-e138-4189-807d-b7f3d5eb1554 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:15.996319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 835/1656] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:17.478885 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a6bcb4-aa52-442d-a346-d9c9d662212e service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:17.481742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b6a6bcb4-aa52-442d-a346-d9c9d662212e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:17.523747 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a6bcb4-aa52-442d-a346-d9c9d662212e service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:17.552964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abef208-7977-4104-b281-5fb6882c55ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:17.553926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abef208-7977-4104-b281-5fb6882c55ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:17.554357 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abef208-7977-4104-b281-5fb6882c55ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:17.554774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abef208-7977-4104-b281-5fb6882c55ab admin admin] PATH_INFO: `/v3/projects/b1363dbf0d3d4afe8d1b407483d3c6fe` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:17.558695 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abef208-7977-4104-b281-5fb6882c55ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=b1363dbf0d3d4afe8d1b407483d3c6fe)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:17.562236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abef208-7977-4104-b281-5fb6882c55ab admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:17.650000 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:DELETE=4,keystone:SELECT=95,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:17.683247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1abef208-7977-4104-b281-5fb6882c55ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b1363dbf0d3d4afe8d1b407483d3c6fe', 'request_id': 'req-1abef208-7977-4104-b281-5fb6882c55ab'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:17.699673 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1abef208-7977-4104-b281-5fb6882c55ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b1363dbf0d3d4afe8d1b407483d3c6fe', 'request_id': 'req-1abef208-7977-4104-b281-5fb6882c55ab'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:17.709377 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1abef208-7977-4104-b281-5fb6882c55ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b1363dbf0d3d4afe8d1b407483d3c6fe', 'request_id': 'req-1abef208-7977-4104-b281-5fb6882c55ab'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:17.728991 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1abef208-7977-4104-b281-5fb6882c55ab admin admin] The token cache is being invalidate because project b1363dbf0d3d4afe8d1b407483d3c6fe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:17.729627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1abef208-7977-4104-b281-5fb6882c55ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:17.730682 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1abef208-7977-4104-b281-5fb6882c55ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:17.733090 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 822/1657] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:17 2022] DELETE /identity/v3/projects/b1363dbf0d3d4afe8d1b407483d3c6fe => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:17.738091 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc7e7db-ae32-414c-9e66-3d59ac10ac0c None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:17.739915 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc7e7db-ae32-414c-9e66-3d59ac10ac0c None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:17.740386 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc7e7db-ae32-414c-9e66-3d59ac10ac0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:17.780507 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-9cc7e7db-ae32-414c-9e66-3d59ac10ac0c None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:17.783269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-9cc7e7db-ae32-414c-9e66-3d59ac10ac0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:17.789197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abef208-7977-4104-b281-5fb6882c55ab admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:17.791322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1abef208-7977-4104-b281-5fb6882c55ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:17.972030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:18.042059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abef208-7977-4104-b281-5fb6882c55ab admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:18.046710 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 836/1658] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:26:18.048464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4caedb8-9040-47b2-bd29-09c135089b7c None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:18.048757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4caedb8-9040-47b2-bd29-09c135089b7c None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:18.049029 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4caedb8-9040-47b2-bd29-09c135089b7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:18.057182 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1605ddf5-f613-434f-8660-fdb6f4ac2953 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:18.057858 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1605ddf5-f613-434f-8660-fdb6f4ac2953 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:18.058397 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1605ddf5-f613-434f-8660-fdb6f4ac2953 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:18.058683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1605ddf5-f613-434f-8660-fdb6f4ac2953 admin admin] PATH_INFO: `/v3/users/e7284a7792a04e9493c54eae58ca89a6` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:18.061242 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1605ddf5-f613-434f-8660-fdb6f4ac2953 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e7284a7792a04e9493c54eae58ca89a6)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:18.062935 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1605ddf5-f613-434f-8660-fdb6f4ac2953 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:18.099116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-e4caedb8-9040-47b2-bd29-09c135089b7c None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:18.106008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e4caedb8-9040-47b2-bd29-09c135089b7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:18.150311 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 837/1659] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:18.151770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4caedb8-9040-47b2-bd29-09c135089b7c None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:18.187952 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4caedb8-9040-47b2-bd29-09c135089b7c None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:18.193517 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8246c7-7a88-4d5b-b277-784d9d8d571c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:18.194140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8246c7-7a88-4d5b-b277-784d9d8d571c admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:18.194719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8246c7-7a88-4d5b-b277-784d9d8d571c admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:18.195283 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8246c7-7a88-4d5b-b277-784d9d8d571c admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:18.196337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8246c7-7a88-4d5b-b277-784d9d8d571c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:18.196826 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8246c7-7a88-4d5b-b277-784d9d8d571c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:18.199017 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8246c7-7a88-4d5b-b277-784d9d8d571c admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:18.210181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1605ddf5-f613-434f-8660-fdb6f4ac2953 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e7284a7792a04e9493c54eae58ca89a6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:18.216606 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 838/1660] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:26:18.222277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8246c7-7a88-4d5b-b277-784d9d8d571c admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:18.236982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1605ddf5-f613-434f-8660-fdb6f4ac2953 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e7284a7792a04e9493c54eae58ca89a6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:18.241317 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8246c7-7a88-4d5b-b277-784d9d8d571c admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:18.248994 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1605ddf5-f613-434f-8660-fdb6f4ac2953 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e7284a7792a04e9493c54eae58ca89a6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:18.257493 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4543f3ce-e76b-4886-bfb8-c84c663db37f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:18.259850 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4543f3ce-e76b-4886-bfb8-c84c663db37f admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:18.260358 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4543f3ce-e76b-4886-bfb8-c84c663db37f admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:18.260824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4543f3ce-e76b-4886-bfb8-c84c663db37f admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:18.262078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4543f3ce-e76b-4886-bfb8-c84c663db37f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:18.264613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4543f3ce-e76b-4886-bfb8-c84c663db37f admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:18.267490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1605ddf5-f613-434f-8660-fdb6f4ac2953 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e7284a7792a04e9493c54eae58ca89a6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:18.268970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1605ddf5-f613-434f-8660-fdb6f4ac2953 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e7284a7792a04e9493c54eae58ca89a6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:18.281287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 823/1661] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:17 2022] DELETE /identity/v3/users/e7284a7792a04e9493c54eae58ca89a6 => generated 0 bytes in 493 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:26:18.286208 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea166919-290a-43a2-92ef-6014c915eab1 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:18.286618 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea166919-290a-43a2-92ef-6014c915eab1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:18.286998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea166919-290a-43a2-92ef-6014c915eab1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:18.367758 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 839/1662] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:18 2022] POST /identity/v3/projects => generated 383 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:18.369528 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4543f3ce-e76b-4886-bfb8-c84c663db37f admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:18.372568 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4543f3ce-e76b-4886-bfb8-c84c663db37f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:18.377572 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ea166919-290a-43a2-92ef-6014c915eab1 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:18.380491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ea166919-290a-43a2-92ef-6014c915eab1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:18.485923 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:18.552140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 824/1663] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:18.573195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea166919-290a-43a2-92ef-6014c915eab1 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:18.574531 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4543f3ce-e76b-4886-bfb8-c84c663db37f admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:18.575156 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ea166919-290a-43a2-92ef-6014c915eab1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:18.587406 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de4b40d-1e5a-4522-b4a3-c9a41af6f218 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:18.588135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de4b40d-1e5a-4522-b4a3-c9a41af6f218 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:18.588560 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de4b40d-1e5a-4522-b4a3-c9a41af6f218 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:18.588901 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de4b40d-1e5a-4522-b4a3-c9a41af6f218 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:18.589913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de4b40d-1e5a-4522-b4a3-c9a41af6f218 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:18.601952 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de4b40d-1e5a-4522-b4a3-c9a41af6f218 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:18.653300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 840/1664] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:18 2022] POST /identity/v3/users => generated 292 bytes in 285 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:26:18.654639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de4b40d-1e5a-4522-b4a3-c9a41af6f218 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:18.657385 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4de4b40d-1e5a-4522-b4a3-c9a41af6f218 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:18.684866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de4b40d-1e5a-4522-b4a3-c9a41af6f218 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:18.694395 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d204adbc-594e-45a8-8a3d-88ec18e846fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:18.695475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204adbc-594e-45a8-8a3d-88ec18e846fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:18.696328 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204adbc-594e-45a8-8a3d-88ec18e846fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:18.696990 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204adbc-594e-45a8-8a3d-88ec18e846fa admin admin] PATH_INFO: `/v3/projects/27128e3e85bb4712a8f6ee5622576cf5` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:18.698675 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d204adbc-594e-45a8-8a3d-88ec18e846fa admin admin] RBAC: Authorizing `identity:delete_project(project_id=27128e3e85bb4712a8f6ee5622576cf5)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:18.699979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: WARNING py.warnings [None req-d204adbc-594e-45a8-8a3d-88ec18e846fa admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 04:26:18.699979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: warnings.warn(msg) Aug 30 04:26:18.699979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: Aug 30 04:26:18.701887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d204adbc-594e-45a8-8a3d-88ec18e846fa admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:18.724096 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea166919-290a-43a2-92ef-6014c915eab1 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:18.735421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2220241-a968-4c3a-adc3-ebd01fc65772 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:18.736090 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2220241-a968-4c3a-adc3-ebd01fc65772 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:18.736373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2220241-a968-4c3a-adc3-ebd01fc65772 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:18.736794 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2220241-a968-4c3a-adc3-ebd01fc65772 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:18.764800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2220241-a968-4c3a-adc3-ebd01fc65772 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:18.766420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2220241-a968-4c3a-adc3-ebd01fc65772 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:18.797925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 825/1665] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:26:18.798770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2220241-a968-4c3a-adc3-ebd01fc65772 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:18.805828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d204adbc-594e-45a8-8a3d-88ec18e846fa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '27128e3e85bb4712a8f6ee5622576cf5', 'request_id': 'req-d204adbc-594e-45a8-8a3d-88ec18e846fa'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:18.816929 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d204adbc-594e-45a8-8a3d-88ec18e846fa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '27128e3e85bb4712a8f6ee5622576cf5', 'request_id': 'req-d204adbc-594e-45a8-8a3d-88ec18e846fa'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:18.826760 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2220241-a968-4c3a-adc3-ebd01fc65772 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:18.828747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d204adbc-594e-45a8-8a3d-88ec18e846fa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '27128e3e85bb4712a8f6ee5622576cf5', 'request_id': 'req-d204adbc-594e-45a8-8a3d-88ec18e846fa'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:18.897592 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d204adbc-594e-45a8-8a3d-88ec18e846fa admin admin] The token cache is being invalidate because project 27128e3e85bb4712a8f6ee5622576cf5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:18.898255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d204adbc-594e-45a8-8a3d-88ec18e846fa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:18.900270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d204adbc-594e-45a8-8a3d-88ec18e846fa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:18.905844 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 841/1666] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:18 2022] DELETE /identity/v3/projects/27128e3e85bb4712a8f6ee5622576cf5 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:18.910973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d204adbc-594e-45a8-8a3d-88ec18e846fa admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:18.913348 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d204adbc-594e-45a8-8a3d-88ec18e846fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:18.990793 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:18.993707 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac68bd89-0e4d-4d7b-84ed-aaabdd320258 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:18.994518 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac68bd89-0e4d-4d7b-84ed-aaabdd320258 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:18.994856 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac68bd89-0e4d-4d7b-84ed-aaabdd320258 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:18.995332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac68bd89-0e4d-4d7b-84ed-aaabdd320258 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:18.995998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac68bd89-0e4d-4d7b-84ed-aaabdd320258 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:18.997729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac68bd89-0e4d-4d7b-84ed-aaabdd320258 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:19.023733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 826/1667] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:18 2022] GET /identity/v3/roles => generated 1420 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:19.026060 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac68bd89-0e4d-4d7b-84ed-aaabdd320258 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:19.033539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ac68bd89-0e4d-4d7b-84ed-aaabdd320258 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:19.066715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac68bd89-0e4d-4d7b-84ed-aaabdd320258 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:19.074763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06794d6b-14cf-426a-823c-bd950ccb15d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:19.075414 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06794d6b-14cf-426a-823c-bd950ccb15d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:19.075699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06794d6b-14cf-426a-823c-bd950ccb15d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:19.075978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06794d6b-14cf-426a-823c-bd950ccb15d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:19.076720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06794d6b-14cf-426a-823c-bd950ccb15d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:19.078398 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06794d6b-14cf-426a-823c-bd950ccb15d1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:19.102305 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 827/1668] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:19 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:19.105395 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06794d6b-14cf-426a-823c-bd950ccb15d1 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:19.132099 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d204adbc-594e-45a8-8a3d-88ec18e846fa admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:19.135890 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06794d6b-14cf-426a-823c-bd950ccb15d1 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:19.141882 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e27275-4c69-43ce-addf-652b00b445d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:19.142757 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e27275-4c69-43ce-addf-652b00b445d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:19.143047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e27275-4c69-43ce-addf-652b00b445d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:19.143435 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e27275-4c69-43ce-addf-652b00b445d2 admin admin] PATH_INFO: `/v3/projects/f4586bb7af2347089f33c5d9388e1eee/users/4b7c1381b7d4435ea8efaf004147011d/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:19.148004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a68e9f-6fa5-4028-9fda-8e3f0f633fda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:19.150389 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a68e9f-6fa5-4028-9fda-8e3f0f633fda admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:19.150772 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a68e9f-6fa5-4028-9fda-8e3f0f633fda admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:19.152767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a68e9f-6fa5-4028-9fda-8e3f0f633fda admin admin] PATH_INFO: `/v3/users/3cbbf4426a06411caa8f453d4a27590a` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:19.158719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a68e9f-6fa5-4028-9fda-8e3f0f633fda admin admin] RBAC: Authorizing `identity:delete_user(user_id=3cbbf4426a06411caa8f453d4a27590a)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:19.179190 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a68e9f-6fa5-4028-9fda-8e3f0f633fda admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:19.189458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e27275-4c69-43ce-addf-652b00b445d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f4586bb7af2347089f33c5d9388e1eee, user_id=4b7c1381b7d4435ea8efaf004147011d, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:19.193284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e27275-4c69-43ce-addf-652b00b445d2 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:19.220843 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 828/1669] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:19 2022] PUT /identity/v3/projects/f4586bb7af2347089f33c5d9388e1eee/users/4b7c1381b7d4435ea8efaf004147011d/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:19.232817 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbead02-71b0-42dd-b029-e1254bd6c3c0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:19.233144 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbead02-71b0-42dd-b029-e1254bd6c3c0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:19.233411 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbead02-71b0-42dd-b029-e1254bd6c3c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:19.304770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e7a68e9f-6fa5-4028-9fda-8e3f0f633fda admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3cbbf4426a06411caa8f453d4a27590a'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:19.311546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-2fbead02-71b0-42dd-b029-e1254bd6c3c0 None None] MFA Rules not processed for user `4b7c1381b7d4435ea8efaf004147011d`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:19.312817 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2fbead02-71b0-42dd-b029-e1254bd6c3c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:19.322101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e7a68e9f-6fa5-4028-9fda-8e3f0f633fda admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3cbbf4426a06411caa8f453d4a27590a'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:19.323344 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e7a68e9f-6fa5-4028-9fda-8e3f0f633fda admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3cbbf4426a06411caa8f453d4a27590a'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:19.340852 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e7a68e9f-6fa5-4028-9fda-8e3f0f633fda admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3cbbf4426a06411caa8f453d4a27590a'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:19.341794 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e7a68e9f-6fa5-4028-9fda-8e3f0f633fda admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3cbbf4426a06411caa8f453d4a27590a'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:19.358725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 842/1670] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:18 2022] DELETE /identity/v3/users/3cbbf4426a06411caa8f453d4a27590a => generated 0 bytes in 449 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:19.423927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:19.510505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 829/1671] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:19 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:19.515468 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a68e9f-6fa5-4028-9fda-8e3f0f633fda admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:19.517228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e7a68e9f-6fa5-4028-9fda-8e3f0f633fda admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:19.668113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a68e9f-6fa5-4028-9fda-8e3f0f633fda admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:19.681579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed53c29-463c-4b93-b8d7-1a276632c3e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:19.683882 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed53c29-463c-4b93-b8d7-1a276632c3e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:19.684166 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed53c29-463c-4b93-b8d7-1a276632c3e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:19.684438 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed53c29-463c-4b93-b8d7-1a276632c3e6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:19.685125 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed53c29-463c-4b93-b8d7-1a276632c3e6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:19.691399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed53c29-463c-4b93-b8d7-1a276632c3e6 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:19.745589 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 843/1672] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:19 2022] POST /identity/v3/projects => generated 379 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:19.750305 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbead02-71b0-42dd-b029-e1254bd6c3c0 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:19.862825 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbead02-71b0-42dd-b029-e1254bd6c3c0 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:19.927871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc4c2f7-7a7c-4697-85c6-785a55e0f445 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:19.928503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc4c2f7-7a7c-4697-85c6-785a55e0f445 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:19.928780 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc4c2f7-7a7c-4697-85c6-785a55e0f445 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:19.929064 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc4c2f7-7a7c-4697-85c6-785a55e0f445 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:19.929731 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc4c2f7-7a7c-4697-85c6-785a55e0f445 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:19.931837 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc4c2f7-7a7c-4697-85c6-785a55e0f445 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:19.970293 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 830/1673] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:19 2022] POST /identity/v3/users => generated 296 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:19.973164 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed53c29-463c-4b93-b8d7-1a276632c3e6 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:19.986733 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed53c29-463c-4b93-b8d7-1a276632c3e6 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:19.996307 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a071267-54e1-4005-a580-f68c1f773eda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:19.997785 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a071267-54e1-4005-a580-f68c1f773eda admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:19.998513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a071267-54e1-4005-a580-f68c1f773eda admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:19.998990 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a071267-54e1-4005-a580-f68c1f773eda admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:19.999907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a071267-54e1-4005-a580-f68c1f773eda admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:20.003212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a071267-54e1-4005-a580-f68c1f773eda admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:20.023569 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 844/1674] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:19 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:20.024956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc4c2f7-7a7c-4697-85c6-785a55e0f445 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:20.039660 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc4c2f7-7a7c-4697-85c6-785a55e0f445 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:20.047019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab27781-7483-4b8b-8125-5ce051f3f9c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:20.048163 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab27781-7483-4b8b-8125-5ce051f3f9c5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:20.048816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab27781-7483-4b8b-8125-5ce051f3f9c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:20.049333 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab27781-7483-4b8b-8125-5ce051f3f9c5 admin admin] PATH_INFO: `/v3/projects/73ccb2772b4a42358fd022daab0c5356/users/b71e2b5f9aa348c6baa2f7def68ad306/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:20.081336 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab27781-7483-4b8b-8125-5ce051f3f9c5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=73ccb2772b4a42358fd022daab0c5356, user_id=b71e2b5f9aa348c6baa2f7def68ad306, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:20.085533 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab27781-7483-4b8b-8125-5ce051f3f9c5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:20.135257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 831/1675] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:20 2022] PUT /identity/v3/projects/73ccb2772b4a42358fd022daab0c5356/users/b71e2b5f9aa348c6baa2f7def68ad306/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:20.137471 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a071267-54e1-4005-a580-f68c1f773eda admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:20.168753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a071267-54e1-4005-a580-f68c1f773eda admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:20.182664 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:20.323298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6666ed9f-e617-45ef-92fe-2eb417e9cd12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:20.325431 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6666ed9f-e617-45ef-92fe-2eb417e9cd12 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:20.325939 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6666ed9f-e617-45ef-92fe-2eb417e9cd12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:20.331607 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6666ed9f-e617-45ef-92fe-2eb417e9cd12 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:20.334368 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6666ed9f-e617-45ef-92fe-2eb417e9cd12 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:20.337170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6666ed9f-e617-45ef-92fe-2eb417e9cd12 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:20.362467 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 845/1676] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:20 2022] GET /identity/v3/roles => generated 1420 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:20.364262 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab27781-7483-4b8b-8125-5ce051f3f9c5 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:20.385102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab27781-7483-4b8b-8125-5ce051f3f9c5 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:20.396013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0580723f-f347-4720-bc3a-b7ab930ddfd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:20.396903 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0580723f-f347-4720-bc3a-b7ab930ddfd2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:20.398687 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0580723f-f347-4720-bc3a-b7ab930ddfd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:20.399267 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0580723f-f347-4720-bc3a-b7ab930ddfd2 admin admin] PATH_INFO: `/v3/projects/73ccb2772b4a42358fd022daab0c5356/users/b71e2b5f9aa348c6baa2f7def68ad306/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:20.404444 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0580723f-f347-4720-bc3a-b7ab930ddfd2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=73ccb2772b4a42358fd022daab0c5356, user_id=b71e2b5f9aa348c6baa2f7def68ad306, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:20.409634 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0580723f-f347-4720-bc3a-b7ab930ddfd2 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:20.481447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 832/1677] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:20 2022] PUT /identity/v3/projects/73ccb2772b4a42358fd022daab0c5356/users/b71e2b5f9aa348c6baa2f7def68ad306/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:20.492435 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5dc11b-2eaf-4e5c-b687-4ea63084a288 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:20.492727 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5dc11b-2eaf-4e5c-b687-4ea63084a288 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:20.493016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5dc11b-2eaf-4e5c-b687-4ea63084a288 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:20.543675 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-da5dc11b-2eaf-4e5c-b687-4ea63084a288 None None] MFA Rules not processed for user `b71e2b5f9aa348c6baa2f7def68ad306`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:20.545063 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-da5dc11b-2eaf-4e5c-b687-4ea63084a288 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:20.671947 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 846/1678] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:20 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:20.672844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0580723f-f347-4720-bc3a-b7ab930ddfd2 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:20.674716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0580723f-f347-4720-bc3a-b7ab930ddfd2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:20.698207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5dc11b-2eaf-4e5c-b687-4ea63084a288 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:20.701177 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-da5dc11b-2eaf-4e5c-b687-4ea63084a288 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:20.757591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:20.812657 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0580723f-f347-4720-bc3a-b7ab930ddfd2 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:20.835619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed7702c-4c80-4b9d-b053-32bafb604a0d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:20.848561 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed7702c-4c80-4b9d-b053-32bafb604a0d service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:20.848906 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed7702c-4c80-4b9d-b053-32bafb604a0d service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:20.849156 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed7702c-4c80-4b9d-b053-32bafb604a0d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:20.861534 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5dc11b-2eaf-4e5c-b687-4ea63084a288 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:20.875445 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3d439a-a083-4d71-9bb8-adfa7e412347 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:20.875650 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed7702c-4c80-4b9d-b053-32bafb604a0d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:20.876111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3d439a-a083-4d71-9bb8-adfa7e412347 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:20.876390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3d439a-a083-4d71-9bb8-adfa7e412347 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:20.876665 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3d439a-a083-4d71-9bb8-adfa7e412347 admin admin] PATH_INFO: `/v3/projects/e35dc7910f7448f1b0646bb291261be0` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:20.877336 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed7702c-4c80-4b9d-b053-32bafb604a0d service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:20.878391 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3d439a-a083-4d71-9bb8-adfa7e412347 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e35dc7910f7448f1b0646bb291261be0)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:20.882576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3d439a-a083-4d71-9bb8-adfa7e412347 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:20.920108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 833/1679] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:20 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 248 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:26:21.007595 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:21.035645 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-2c3d439a-a083-4d71-9bb8-adfa7e412347 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e35dc7910f7448f1b0646bb291261be0', 'request_id': 'req-2c3d439a-a083-4d71-9bb8-adfa7e412347'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:21.053165 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-2c3d439a-a083-4d71-9bb8-adfa7e412347 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e35dc7910f7448f1b0646bb291261be0', 'request_id': 'req-2c3d439a-a083-4d71-9bb8-adfa7e412347'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:21.067686 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed7702c-4c80-4b9d-b053-32bafb604a0d service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:21.070025 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5ed7702c-4c80-4b9d-b053-32bafb604a0d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:21.074984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-2c3d439a-a083-4d71-9bb8-adfa7e412347 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e35dc7910f7448f1b0646bb291261be0', 'request_id': 'req-2c3d439a-a083-4d71-9bb8-adfa7e412347'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:21.109320 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed7702c-4c80-4b9d-b053-32bafb604a0d service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:21.109828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-2c3d439a-a083-4d71-9bb8-adfa7e412347 admin admin] The token cache is being invalidate because project e35dc7910f7448f1b0646bb291261be0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:21.110132 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-2c3d439a-a083-4d71-9bb8-adfa7e412347 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:21.112337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-2c3d439a-a083-4d71-9bb8-adfa7e412347 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:21.122637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 847/1680] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:20 2022] DELETE /identity/v3/projects/e35dc7910f7448f1b0646bb291261be0 => generated 0 bytes in 422 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:26:21.134342 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102d5658-674d-4466-9b05-d98cd5a4b7c0 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:21.134832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102d5658-674d-4466-9b05-d98cd5a4b7c0 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:21.135113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102d5658-674d-4466-9b05-d98cd5a4b7c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:21.221086 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-102d5658-674d-4466-9b05-d98cd5a4b7c0 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:21.224092 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-102d5658-674d-4466-9b05-d98cd5a4b7c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:21.281813 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3441d38d-3043-4328-b697-2b6dc39dbce7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:21.282487 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3441d38d-3043-4328-b697-2b6dc39dbce7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:21.282768 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3441d38d-3043-4328-b697-2b6dc39dbce7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:21.283042 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3441d38d-3043-4328-b697-2b6dc39dbce7 admin admin] PATH_INFO: `/v3/projects/21c540078fc04f65b8e365c978b333cd` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:21.298105 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3441d38d-3043-4328-b697-2b6dc39dbce7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=21c540078fc04f65b8e365c978b333cd)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:21.300245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3441d38d-3043-4328-b697-2b6dc39dbce7 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:21.417470 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 848/1681] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:21.417470 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102d5658-674d-4466-9b05-d98cd5a4b7c0 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:21.418619 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-102d5658-674d-4466-9b05-d98cd5a4b7c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:21.419156 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3441d38d-3043-4328-b697-2b6dc39dbce7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '21c540078fc04f65b8e365c978b333cd', 'request_id': 'req-3441d38d-3043-4328-b697-2b6dc39dbce7'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:21.437384 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3441d38d-3043-4328-b697-2b6dc39dbce7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '21c540078fc04f65b8e365c978b333cd', 'request_id': 'req-3441d38d-3043-4328-b697-2b6dc39dbce7'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:21.455654 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3441d38d-3043-4328-b697-2b6dc39dbce7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '21c540078fc04f65b8e365c978b333cd', 'request_id': 'req-3441d38d-3043-4328-b697-2b6dc39dbce7'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:21.481686 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3441d38d-3043-4328-b697-2b6dc39dbce7 admin admin] The token cache is being invalidate because project 21c540078fc04f65b8e365c978b333cd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:21.481997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3441d38d-3043-4328-b697-2b6dc39dbce7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:21.499785 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3441d38d-3043-4328-b697-2b6dc39dbce7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:21.504958 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 834/1682] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:21 2022] DELETE /identity/v3/projects/21c540078fc04f65b8e365c978b333cd => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:21.506301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0833443-1979-49cf-b68f-6d06597fe565 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:21.506580 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0833443-1979-49cf-b68f-6d06597fe565 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:21.506861 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0833443-1979-49cf-b68f-6d06597fe565 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:21.605075 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102d5658-674d-4466-9b05-d98cd5a4b7c0 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:21.630021 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c0833443-1979-49cf-b68f-6d06597fe565 None None] MFA Rules not processed for user `cbf574ed57d446bc8fabe7813e64e92c`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:21.631377 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c0833443-1979-49cf-b68f-6d06597fe565 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:21.735967 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2053321a-759e-42ca-979f-13fe12908ddb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:21.736561 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2053321a-759e-42ca-979f-13fe12908ddb service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:21.736836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2053321a-759e-42ca-979f-13fe12908ddb service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:21.737191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2053321a-759e-42ca-979f-13fe12908ddb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:21.742105 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-2053321a-759e-42ca-979f-13fe12908ddb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:21.823609 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:21.824497 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:21.841402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 835/1683] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:21 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 336 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:21.844672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6eac9c-1eb0-4244-b739-4d8ee4e6924c None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:21.845586 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6eac9c-1eb0-4244-b739-4d8ee4e6924c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:21.846314 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6eac9c-1eb0-4244-b739-4d8ee4e6924c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:21.893547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2053321a-759e-42ca-979f-13fe12908ddb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:21.895228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2053321a-759e-42ca-979f-13fe12908ddb service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:21.915449 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 849/1684] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:21 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 503 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:21.921657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd31974c-8254-4c25-bf01-465aba2e0783 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:21.922310 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd31974c-8254-4c25-bf01-465aba2e0783 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:21.922589 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd31974c-8254-4c25-bf01-465aba2e0783 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:21.927598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-1d6eac9c-1eb0-4244-b739-4d8ee4e6924c None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:21.928826 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1d6eac9c-1eb0-4244-b739-4d8ee4e6924c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:21.965927 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-fd31974c-8254-4c25-bf01-465aba2e0783 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:21.967328 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-fd31974c-8254-4c25-bf01-465aba2e0783 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:22.076869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 836/1685] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:22.078057 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32529be-9d9a-4adc-b0d3-8ae7f03ac63d None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:22.078346 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32529be-9d9a-4adc-b0d3-8ae7f03ac63d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:22.078619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32529be-9d9a-4adc-b0d3-8ae7f03ac63d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:22.138394 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c32529be-9d9a-4adc-b0d3-8ae7f03ac63d None None] MFA Rules not processed for user `cbf574ed57d446bc8fabe7813e64e92c`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:22.139612 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c32529be-9d9a-4adc-b0d3-8ae7f03ac63d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:22.149659 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 850/1686] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:22.149659 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd31974c-8254-4c25-bf01-465aba2e0783 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:22.168282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 837/1687] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:22 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:22.169626 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebd9ab2-5a42-447f-8ef3-30048d05f772 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:22.169907 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebd9ab2-5a42-447f-8ef3-30048d05f772 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:22.170176 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebd9ab2-5a42-447f-8ef3-30048d05f772 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:22.180607 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd31974c-8254-4c25-bf01-465aba2e0783 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:22.193657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc2b685-be2d-4e46-bab0-2e1e32599300 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:22.194129 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc2b685-be2d-4e46-bab0-2e1e32599300 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:22.194413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc2b685-be2d-4e46-bab0-2e1e32599300 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:22.194683 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc2b685-be2d-4e46-bab0-2e1e32599300 admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:22.195346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc2b685-be2d-4e46-bab0-2e1e32599300 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:22.195646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc2b685-be2d-4e46-bab0-2e1e32599300 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:22.197704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc2b685-be2d-4e46-bab0-2e1e32599300 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:22.226391 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 851/1688] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:26:22.227548 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-1ebd9ab2-5a42-447f-8ef3-30048d05f772 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:22.228989 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1ebd9ab2-5a42-447f-8ef3-30048d05f772 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:22.232004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc2b685-be2d-4e46-bab0-2e1e32599300 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:22.235283 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-6fc2b685-be2d-4e46-bab0-2e1e32599300 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:22.248201 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 838/1689] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:22.253230 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebd9ab2-5a42-447f-8ef3-30048d05f772 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:22.276603 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebd9ab2-5a42-447f-8ef3-30048d05f772 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:22.292803 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d288e500-1677-4811-9a82-b794bfa2d0db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:22.293664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d288e500-1677-4811-9a82-b794bfa2d0db admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:22.294097 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d288e500-1677-4811-9a82-b794bfa2d0db admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:22.294504 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d288e500-1677-4811-9a82-b794bfa2d0db admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:22.295352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d288e500-1677-4811-9a82-b794bfa2d0db admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:22.301484 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d288e500-1677-4811-9a82-b794bfa2d0db admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:22.372498 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 839/1690] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:22 2022] POST /identity/v3/projects => generated 405 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:22.373504 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d288e500-1677-4811-9a82-b794bfa2d0db admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:22.400787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc2b685-be2d-4e46-bab0-2e1e32599300 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:22.465603 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d288e500-1677-4811-9a82-b794bfa2d0db admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:22.486527 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804330fc-9071-4f4d-8ac2-1757fa2b4a8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:22.487204 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804330fc-9071-4f4d-8ac2-1757fa2b4a8e admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:22.487490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804330fc-9071-4f4d-8ac2-1757fa2b4a8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:22.487815 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804330fc-9071-4f4d-8ac2-1757fa2b4a8e admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:22.488534 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804330fc-9071-4f4d-8ac2-1757fa2b4a8e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:22.488784 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804330fc-9071-4f4d-8ac2-1757fa2b4a8e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:22.492991 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804330fc-9071-4f4d-8ac2-1757fa2b4a8e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:22.514562 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:22.518305 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 840/1691] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:26:22.524383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e0aaaf-a3f8-44c3-993f-b430454b6e60 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:22.524676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e0aaaf-a3f8-44c3-993f-b430454b6e60 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:22.525022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e0aaaf-a3f8-44c3-993f-b430454b6e60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:22.541182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05417f2c-88c2-46a5-be24-8ed294839611 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:22.541835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05417f2c-88c2-46a5-be24-8ed294839611 service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:22.542121 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05417f2c-88c2-46a5-be24-8ed294839611 service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:22.542397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05417f2c-88c2-46a5-be24-8ed294839611 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:22.588585 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-48e0aaaf-a3f8-44c3-993f-b430454b6e60 None None] MFA Rules not processed for user `8ef7a83a87d44fc0a5647cb60057c036`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:22.590490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-48e0aaaf-a3f8-44c3-993f-b430454b6e60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:22.651533 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05417f2c-88c2-46a5-be24-8ed294839611 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:22.653258 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05417f2c-88c2-46a5-be24-8ed294839611 service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:22.673965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 852/1692] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:22 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:22.674843 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05417f2c-88c2-46a5-be24-8ed294839611 service nova] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:22.682382 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:22.697220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05417f2c-88c2-46a5-be24-8ed294839611 service nova] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:22.712376 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0553af-c0fd-43df-8a01-99e57cc0af1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:22.713196 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0553af-c0fd-43df-8a01-99e57cc0af1e admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:22.713482 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0553af-c0fd-43df-8a01-99e57cc0af1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:22.713756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0553af-c0fd-43df-8a01-99e57cc0af1e admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:22.715139 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0553af-c0fd-43df-8a01-99e57cc0af1e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:22.716369 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0553af-c0fd-43df-8a01-99e57cc0af1e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:22.730972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 841/1693] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:22 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:22.731857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e0aaaf-a3f8-44c3-993f-b430454b6e60 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:22.751267 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e0aaaf-a3f8-44c3-993f-b430454b6e60 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:22.764057 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7c8e97-2903-46da-8c12-55698ae719b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:22.766053 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7c8e97-2903-46da-8c12-55698ae719b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:22.767569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7c8e97-2903-46da-8c12-55698ae719b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:22.767979 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7c8e97-2903-46da-8c12-55698ae719b2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:22.768746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7c8e97-2903-46da-8c12-55698ae719b2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:22.773473 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 853/1694] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:22 2022] POST /identity/v3/users => generated 303 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:22.775672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7c8e97-2903-46da-8c12-55698ae719b2 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:22.776257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0553af-c0fd-43df-8a01-99e57cc0af1e admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:22.783677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-5d0553af-c0fd-43df-8a01-99e57cc0af1e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:22.819487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0553af-c0fd-43df-8a01-99e57cc0af1e admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:22.842307 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 842/1695] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:22 2022] POST /identity/v3/projects => generated 355 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:22.843180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8442aba-9907-4cff-865f-9d626f1277d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:22.843848 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8442aba-9907-4cff-865f-9d626f1277d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:22.844599 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7c8e97-2903-46da-8c12-55698ae719b2 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:22.846392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8442aba-9907-4cff-865f-9d626f1277d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:22.846848 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8442aba-9907-4cff-865f-9d626f1277d0 admin admin] PATH_INFO: `/v3/users/4b7c1381b7d4435ea8efaf004147011d` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:22.851616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0e7c8e97-2903-46da-8c12-55698ae719b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:22.851753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8442aba-9907-4cff-865f-9d626f1277d0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4b7c1381b7d4435ea8efaf004147011d)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:22.856485 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8442aba-9907-4cff-865f-9d626f1277d0 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:22.951930 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c8442aba-9907-4cff-865f-9d626f1277d0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4b7c1381b7d4435ea8efaf004147011d'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:22.965986 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c8442aba-9907-4cff-865f-9d626f1277d0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4b7c1381b7d4435ea8efaf004147011d'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:22.967134 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c8442aba-9907-4cff-865f-9d626f1277d0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4b7c1381b7d4435ea8efaf004147011d'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:22.984771 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c8442aba-9907-4cff-865f-9d626f1277d0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4b7c1381b7d4435ea8efaf004147011d'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:22.985735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c8442aba-9907-4cff-865f-9d626f1277d0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4b7c1381b7d4435ea8efaf004147011d'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:23.000811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7c8e97-2903-46da-8c12-55698ae719b2 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:23.007643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 854/1696] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:22 2022] DELETE /identity/v3/users/4b7c1381b7d4435ea8efaf004147011d => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:23.009205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3e7a80-0824-42d9-90f0-43faea19048b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:23.009540 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8442aba-9907-4cff-865f-9d626f1277d0 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:23.010822 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3e7a80-0824-42d9-90f0-43faea19048b service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:23.011147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3e7a80-0824-42d9-90f0-43faea19048b service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:23.011254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c8442aba-9907-4cff-865f-9d626f1277d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:23.011686 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3e7a80-0824-42d9-90f0-43faea19048b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:23.013841 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2b3e7a80-0824-42d9-90f0-43faea19048b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:23.095840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:23.129497 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8442aba-9907-4cff-865f-9d626f1277d0 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:23.130402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3e7a80-0824-42d9-90f0-43faea19048b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:23.136162 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280b6b20-2ce3-423a-893b-c9111071b142 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:23.137343 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280b6b20-2ce3-423a-893b-c9111071b142 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:23.137807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280b6b20-2ce3-423a-893b-c9111071b142 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:23.138273 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280b6b20-2ce3-423a-893b-c9111071b142 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:23.139259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280b6b20-2ce3-423a-893b-c9111071b142 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:23.141330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280b6b20-2ce3-423a-893b-c9111071b142 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:23.143690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3e7a80-0824-42d9-90f0-43faea19048b service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:23.160331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 855/1697] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:23 2022] GET /identity/v3/roles => generated 1420 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:23.162616 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280b6b20-2ce3-423a-893b-c9111071b142 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:23.164848 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-280b6b20-2ce3-423a-893b-c9111071b142 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:23.168033 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 843/1698] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:22 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:23.182346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280b6b20-2ce3-423a-893b-c9111071b142 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:23.184375 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69557155-1d5a-498b-95fb-03d03fa222f4 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:23.184887 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69557155-1d5a-498b-95fb-03d03fa222f4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:23.185819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69557155-1d5a-498b-95fb-03d03fa222f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:23.193410 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b61f59-4f5f-4126-a7c6-5fe0c8164d4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:23.194429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b61f59-4f5f-4126-a7c6-5fe0c8164d4e admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:23.194921 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b61f59-4f5f-4126-a7c6-5fe0c8164d4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:23.195377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b61f59-4f5f-4126-a7c6-5fe0c8164d4e admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:23.196594 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b61f59-4f5f-4126-a7c6-5fe0c8164d4e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:23.199176 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b61f59-4f5f-4126-a7c6-5fe0c8164d4e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:23.240011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 856/1699] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:23 2022] POST /identity/v3/users => generated 278 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:23.241712 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b61f59-4f5f-4126-a7c6-5fe0c8164d4e admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:23.242001 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-69557155-1d5a-498b-95fb-03d03fa222f4 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:23.247239 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-69557155-1d5a-498b-95fb-03d03fa222f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:23.262062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b61f59-4f5f-4126-a7c6-5fe0c8164d4e admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:23.276012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8107952-4ea0-4047-85b4-d4c12e4c303e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:23.276371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 844/1700] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:23.277356 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69557155-1d5a-498b-95fb-03d03fa222f4 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:23.278311 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8107952-4ea0-4047-85b4-d4c12e4c303e admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:23.280667 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8107952-4ea0-4047-85b4-d4c12e4c303e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:23.280667 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8107952-4ea0-4047-85b4-d4c12e4c303e admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:23.281681 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8107952-4ea0-4047-85b4-d4c12e4c303e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:23.288195 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8107952-4ea0-4047-85b4-d4c12e4c303e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:23.312400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69557155-1d5a-498b-95fb-03d03fa222f4 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:23.323920 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 857/1701] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:23 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:23.325316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8107952-4ea0-4047-85b4-d4c12e4c303e admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:23.326980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e3b9af-b04b-49eb-b19b-ab78904a24ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:23.332795 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e3b9af-b04b-49eb-b19b-ab78904a24ce admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:23.333224 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e3b9af-b04b-49eb-b19b-ab78904a24ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:23.333505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e3b9af-b04b-49eb-b19b-ab78904a24ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:23.334187 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e3b9af-b04b-49eb-b19b-ab78904a24ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:23.343851 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e3b9af-b04b-49eb-b19b-ab78904a24ce admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:23.364107 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8107952-4ea0-4047-85b4-d4c12e4c303e admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:23.385886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d1e372-a92e-4b23-8187-b0c23c25be36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:23.386948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d1e372-a92e-4b23-8187-b0c23c25be36 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:23.387300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d1e372-a92e-4b23-8187-b0c23c25be36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:23.393381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d1e372-a92e-4b23-8187-b0c23c25be36 admin admin] PATH_INFO: `/v3/projects/8ed8492885cb49dc93bedfe390643141/users/88a5946efd374183b16197f499e2dfa3/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:23.397475 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 845/1702] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:23 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:23.400472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e3b9af-b04b-49eb-b19b-ab78904a24ce admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:23.417132 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e3b9af-b04b-49eb-b19b-ab78904a24ce admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:23.426986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c297885d-0f88-4fc5-bce8-e0acd7cb70dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:23.428160 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c297885d-0f88-4fc5-bce8-e0acd7cb70dd admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:23.428495 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c297885d-0f88-4fc5-bce8-e0acd7cb70dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:23.428962 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c297885d-0f88-4fc5-bce8-e0acd7cb70dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:23.429911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c297885d-0f88-4fc5-bce8-e0acd7cb70dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:23.432019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c297885d-0f88-4fc5-bce8-e0acd7cb70dd admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:23.436946 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d1e372-a92e-4b23-8187-b0c23c25be36 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ed8492885cb49dc93bedfe390643141, user_id=88a5946efd374183b16197f499e2dfa3, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:23.444066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d1e372-a92e-4b23-8187-b0c23c25be36 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:23.456688 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 846/1703] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:23 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:23.458375 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c297885d-0f88-4fc5-bce8-e0acd7cb70dd admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:23.461174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c297885d-0f88-4fc5-bce8-e0acd7cb70dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:23.492298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 858/1704] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:23 2022] PUT /identity/v3/projects/8ed8492885cb49dc93bedfe390643141/users/88a5946efd374183b16197f499e2dfa3/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:23.494379 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d1e372-a92e-4b23-8187-b0c23c25be36 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:23.538743 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:23.587367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c297885d-0f88-4fc5-bce8-e0acd7cb70dd admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:23.677024 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d1e372-a92e-4b23-8187-b0c23c25be36 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:23.687373 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f155e3a7-cecf-4f8a-b55a-c113f426888c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:23.688067 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f155e3a7-cecf-4f8a-b55a-c113f426888c admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:23.690218 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f155e3a7-cecf-4f8a-b55a-c113f426888c admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:23.692043 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486bb0ed-6632-4794-b5be-08094ea4a641 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:23.692591 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f155e3a7-cecf-4f8a-b55a-c113f426888c admin admin] PATH_INFO: `/v3/projects/06943f9f571c4d3caa60cba6516e18a2/users/e1f89af210e1447aab194bf6bc61cc32/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:23.693765 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486bb0ed-6632-4794-b5be-08094ea4a641 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:23.694234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486bb0ed-6632-4794-b5be-08094ea4a641 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:23.694691 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486bb0ed-6632-4794-b5be-08094ea4a641 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:23.714872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486bb0ed-6632-4794-b5be-08094ea4a641 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:23.716625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486bb0ed-6632-4794-b5be-08094ea4a641 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:23.734422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f155e3a7-cecf-4f8a-b55a-c113f426888c admin admin] RBAC: Authorizing `identity:create_grant(project_id=06943f9f571c4d3caa60cba6516e18a2, user_id=e1f89af210e1447aab194bf6bc61cc32, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:23.736296 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f155e3a7-cecf-4f8a-b55a-c113f426888c admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:23.757909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 847/1705] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:23.759836 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e096cfd-c11f-423a-b8d5-1551767de2f0 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:23.760327 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e096cfd-c11f-423a-b8d5-1551767de2f0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:23.760828 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e096cfd-c11f-423a-b8d5-1551767de2f0 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:23.762784 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 848/1706] 10.222.0.85 () {54 vars in 912 bytes} [Tue Aug 30 04:26:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:26:23.767209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d655c27-74ae-4fd5-80ce-955428d975ac None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:23.767815 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d655c27-74ae-4fd5-80ce-955428d975ac None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:23.768309 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d655c27-74ae-4fd5-80ce-955428d975ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:23.781754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 859/1707] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:23 2022] PUT /identity/v3/projects/06943f9f571c4d3caa60cba6516e18a2/users/e1f89af210e1447aab194bf6bc61cc32/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:23.785208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6f3118-1548-4024-89cb-af302520dea8 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:23.785208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6f3118-1548-4024-89cb-af302520dea8 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:23.785208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6f3118-1548-4024-89cb-af302520dea8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:23.798807 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-7d655c27-74ae-4fd5-80ce-955428d975ac None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:23.802207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7d655c27-74ae-4fd5-80ce-955428d975ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:23.874392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-cd6f3118-1548-4024-89cb-af302520dea8 None None] MFA Rules not processed for user `5c2edd77896a4534888d6758ec0ee0bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:23.876136 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-cd6f3118-1548-4024-89cb-af302520dea8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:23.891049 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 849/1708] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:23.892854 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27df73fb-6b1c-4b70-af8e-562521565895 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:23.895004 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27df73fb-6b1c-4b70-af8e-562521565895 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:23.895357 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27df73fb-6b1c-4b70-af8e-562521565895 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:23.956824 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-27df73fb-6b1c-4b70-af8e-562521565895 None None] MFA Rules not processed for user `e1f89af210e1447aab194bf6bc61cc32`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:23.958443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-27df73fb-6b1c-4b70-af8e-562521565895 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:24.032454 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 860/1709] 10.222.0.85 () {58 vars in 1009 bytes} [Tue Aug 30 04:26:23 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:24.040442 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6f3118-1548-4024-89cb-af302520dea8 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:24.056013 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-cd6f3118-1548-4024-89cb-af302520dea8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:24.088741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6f3118-1548-4024-89cb-af302520dea8 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:24.110068 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1877f8-e815-4ae6-9e9b-947b1232fa99 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:24.110869 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1877f8-e815-4ae6-9e9b-947b1232fa99 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:24.111159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1877f8-e815-4ae6-9e9b-947b1232fa99 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:24.111437 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1877f8-e815-4ae6-9e9b-947b1232fa99 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:24.150031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1877f8-e815-4ae6-9e9b-947b1232fa99 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:24.151513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1877f8-e815-4ae6-9e9b-947b1232fa99 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:24.156307 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:24.168717 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 850/1710] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:23 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:24.169797 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27df73fb-6b1c-4b70-af8e-562521565895 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:24.183251 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 861/1711] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:24.188088 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1877f8-e815-4ae6-9e9b-947b1232fa99 service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:24.192253 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27df73fb-6b1c-4b70-af8e-562521565895 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:24.198133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd04a6c-7de7-4cc8-a116-3e8e9a25424c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:24.198786 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd04a6c-7de7-4cc8-a116-3e8e9a25424c service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:24.199067 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd04a6c-7de7-4cc8-a116-3e8e9a25424c service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:24.199343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd04a6c-7de7-4cc8-a116-3e8e9a25424c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:24.210836 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd04a6c-7de7-4cc8-a116-3e8e9a25424c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:24.215310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd04a6c-7de7-4cc8-a116-3e8e9a25424c service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:24.225883 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1877f8-e815-4ae6-9e9b-947b1232fa99 service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:24.236865 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb0d82f-6b45-4922-b733-cd573e3f098e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:24.238307 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb0d82f-6b45-4922-b733-cd573e3f098e admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:24.238600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb0d82f-6b45-4922-b733-cd573e3f098e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:24.238873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb0d82f-6b45-4922-b733-cd573e3f098e admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:24.242530 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 851/1712] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:24 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 73 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:24.249429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb0d82f-6b45-4922-b733-cd573e3f098e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:24.251464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb0d82f-6b45-4922-b733-cd573e3f098e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:24.352346 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd04a6c-7de7-4cc8-a116-3e8e9a25424c service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:24.360720 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 862/1713] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:24 2022] POST /identity/v3/projects => generated 357 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:24.391402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd04a6c-7de7-4cc8-a116-3e8e9a25424c service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:24.417155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:24.569162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c063de41-ac95-4aea-b32c-1eaf92170952 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:24.570076 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c063de41-ac95-4aea-b32c-1eaf92170952 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:24.570076 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c063de41-ac95-4aea-b32c-1eaf92170952 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:24.571747 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c063de41-ac95-4aea-b32c-1eaf92170952 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:24.572423 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c063de41-ac95-4aea-b32c-1eaf92170952 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:24.574416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c063de41-ac95-4aea-b32c-1eaf92170952 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:24.647191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 852/1714] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:24 2022] POST /identity/v3/users => generated 285 bytes in 295 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:26:24.651662 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb0d82f-6b45-4922-b733-cd573e3f098e admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:24.664761 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb0d82f-6b45-4922-b733-cd573e3f098e admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:24.671502 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63a20ff-ac80-4355-ba3c-07ac937c76fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:24.672311 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63a20ff-ac80-4355-ba3c-07ac937c76fc admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:24.672619 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63a20ff-ac80-4355-ba3c-07ac937c76fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:24.672937 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63a20ff-ac80-4355-ba3c-07ac937c76fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:24.673702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63a20ff-ac80-4355-ba3c-07ac937c76fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:24.676044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63a20ff-ac80-4355-ba3c-07ac937c76fc admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:24.709440 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c063de41-ac95-4aea-b32c-1eaf92170952 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:24.721512 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 863/1715] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:24 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:24.726458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c063de41-ac95-4aea-b32c-1eaf92170952 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:24.736986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f88b11-976c-4ece-9b8d-428fd52418a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:24.740415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f88b11-976c-4ece-9b8d-428fd52418a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:24.740415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f88b11-976c-4ece-9b8d-428fd52418a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:24.740415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f88b11-976c-4ece-9b8d-428fd52418a4 admin admin] PATH_INFO: `/v3/projects/40bf5a94be374c9e8bbba3918c8b0c7d/users/acc83508c5c145bd84a33181afc9a0c2/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:24.792612 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f88b11-976c-4ece-9b8d-428fd52418a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=40bf5a94be374c9e8bbba3918c8b0c7d, user_id=acc83508c5c145bd84a33181afc9a0c2, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:24.803732 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f88b11-976c-4ece-9b8d-428fd52418a4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:24.848836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63a20ff-ac80-4355-ba3c-07ac937c76fc admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:24.854341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 853/1716] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:24 2022] PUT /identity/v3/projects/40bf5a94be374c9e8bbba3918c8b0c7d/users/acc83508c5c145bd84a33181afc9a0c2/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:26:24.861843 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63a20ff-ac80-4355-ba3c-07ac937c76fc admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:25.073329 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d48e3a-75d4-40d8-a2c7-4423df9fc2ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:25.073965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d48e3a-75d4-40d8-a2c7-4423df9fc2ab admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:25.074252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d48e3a-75d4-40d8-a2c7-4423df9fc2ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:25.074528 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d48e3a-75d4-40d8-a2c7-4423df9fc2ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:25.075191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d48e3a-75d4-40d8-a2c7-4423df9fc2ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:25.076927 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d48e3a-75d4-40d8-a2c7-4423df9fc2ab admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:25.139375 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 864/1717] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:24 2022] GET /identity/v3/roles => generated 1420 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:25.142973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f88b11-976c-4ece-9b8d-428fd52418a4 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:25.167110 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f88b11-976c-4ece-9b8d-428fd52418a4 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:25.188268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ca752d-b2a2-4cd2-831f-c43c53637b46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:25.188942 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ca752d-b2a2-4cd2-831f-c43c53637b46 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:25.189332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ca752d-b2a2-4cd2-831f-c43c53637b46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:25.189619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ca752d-b2a2-4cd2-831f-c43c53637b46 admin admin] PATH_INFO: `/v3/projects/40bf5a94be374c9e8bbba3918c8b0c7d/users/acc83508c5c145bd84a33181afc9a0c2/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:25.198797 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ca752d-b2a2-4cd2-831f-c43c53637b46 admin admin] RBAC: Authorizing `identity:create_grant(project_id=40bf5a94be374c9e8bbba3918c8b0c7d, user_id=acc83508c5c145bd84a33181afc9a0c2, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:25.201941 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ca752d-b2a2-4cd2-831f-c43c53637b46 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:25.225288 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 854/1718] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:25 2022] PUT /identity/v3/projects/40bf5a94be374c9e8bbba3918c8b0c7d/users/acc83508c5c145bd84a33181afc9a0c2/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:25.236044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b41a4d7-11fe-46f1-be11-4f5f86d08dcc None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:25.236712 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b41a4d7-11fe-46f1-be11-4f5f86d08dcc None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:25.236712 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b41a4d7-11fe-46f1-be11-4f5f86d08dcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:25.319682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-2b41a4d7-11fe-46f1-be11-4f5f86d08dcc None None] MFA Rules not processed for user `acc83508c5c145bd84a33181afc9a0c2`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:25.339790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-2b41a4d7-11fe-46f1-be11-4f5f86d08dcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:25.591513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 865/1719] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:25 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 356 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:26:25.630501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ca752d-b2a2-4cd2-831f-c43c53637b46 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:25.632458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-13ca752d-b2a2-4cd2-831f-c43c53637b46 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:25.802974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ca752d-b2a2-4cd2-831f-c43c53637b46 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:25.815590 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fd9d96-2a27-4787-80c7-d3d6e149fa81 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:25.816869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fd9d96-2a27-4787-80c7-d3d6e149fa81 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:25.817389 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fd9d96-2a27-4787-80c7-d3d6e149fa81 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:25.817917 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fd9d96-2a27-4787-80c7-d3d6e149fa81 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:25.840709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fd9d96-2a27-4787-80c7-d3d6e149fa81 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:25.842913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fd9d96-2a27-4787-80c7-d3d6e149fa81 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:25.913378 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 855/1720] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:25 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 283 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:26.006718 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b41a4d7-11fe-46f1-be11-4f5f86d08dcc None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:26.008700 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-2b41a4d7-11fe-46f1-be11-4f5f86d08dcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:26.156621 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:26.157500 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872e600c-c88a-48d9-8b09-57b4ea5926dc None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:26.159464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872e600c-c88a-48d9-8b09-57b4ea5926dc None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:26.160236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872e600c-c88a-48d9-8b09-57b4ea5926dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:26.221482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-872e600c-c88a-48d9-8b09-57b4ea5926dc None None] MFA Rules not processed for user `cbf574ed57d446bc8fabe7813e64e92c`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:26.226308 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-872e600c-c88a-48d9-8b09-57b4ea5926dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:26.227590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b41a4d7-11fe-46f1-be11-4f5f86d08dcc None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:26.242755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed15bf7-9a9a-441f-ac8b-afcecc65fbcf service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '359d79cbfcad49beba2b495066824128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:26.249281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed15bf7-9a9a-441f-ac8b-afcecc65fbcf service placement] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:26.249574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed15bf7-9a9a-441f-ac8b-afcecc65fbcf service placement] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:26.249852 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed15bf7-9a9a-441f-ac8b-afcecc65fbcf service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:26.251773 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-bed15bf7-9a9a-441f-ac8b-afcecc65fbcf service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:26.284256 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed15bf7-9a9a-441f-ac8b-afcecc65fbcf service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:26.290801 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed15bf7-9a9a-441f-ac8b-afcecc65fbcf service placement] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:26.326332 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 866/1721] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:26 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:26.330407 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed15bf7-9a9a-441f-ac8b-afcecc65fbcf service placement] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:26.335581 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-bed15bf7-9a9a-441f-ac8b-afcecc65fbcf service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:26.395169 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 856/1722] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:26 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:26.396146 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872e600c-c88a-48d9-8b09-57b4ea5926dc None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:26.413112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872e600c-c88a-48d9-8b09-57b4ea5926dc None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:26.419661 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97dde72-58db-47d9-8595-93bfd1594514 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:26.420779 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97dde72-58db-47d9-8595-93bfd1594514 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:26.421603 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97dde72-58db-47d9-8595-93bfd1594514 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:26.422169 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97dde72-58db-47d9-8595-93bfd1594514 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:26.427974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:26.511090 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed15bf7-9a9a-441f-ac8b-afcecc65fbcf service placement] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:26.527094 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e9e025-e58f-47a1-b056-678848157320 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:26.527756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e9e025-e58f-47a1-b056-678848157320 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:26.528030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e9e025-e58f-47a1-b056-678848157320 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:26.528317 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e9e025-e58f-47a1-b056-678848157320 admin admin] PATH_INFO: `/v3/users/d8444f6398e04554a7907a2d3e0eb1cf` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:26.534816 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e9e025-e58f-47a1-b056-678848157320 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d8444f6398e04554a7907a2d3e0eb1cf)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:26.536859 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e9e025-e58f-47a1-b056-678848157320 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:26.624799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97dde72-58db-47d9-8595-93bfd1594514 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:26.627405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97dde72-58db-47d9-8595-93bfd1594514 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:26.651266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b9e9e025-e58f-47a1-b056-678848157320 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd8444f6398e04554a7907a2d3e0eb1cf'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:26.656881 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 857/1723] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:26 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:26.660834 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6488635b-7f0a-42bb-bd7e-9d4b797212dc None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:26.662571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6488635b-7f0a-42bb-bd7e-9d4b797212dc None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:26.662986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6488635b-7f0a-42bb-bd7e-9d4b797212dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:26.666022 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b9e9e025-e58f-47a1-b056-678848157320 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd8444f6398e04554a7907a2d3e0eb1cf'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:26.668367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b9e9e025-e58f-47a1-b056-678848157320 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd8444f6398e04554a7907a2d3e0eb1cf'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:26.735827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-6488635b-7f0a-42bb-bd7e-9d4b797212dc None None] MFA Rules not processed for user `88a5946efd374183b16197f499e2dfa3`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:26.739299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6488635b-7f0a-42bb-bd7e-9d4b797212dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:26.742426 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b9e9e025-e58f-47a1-b056-678848157320 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd8444f6398e04554a7907a2d3e0eb1cf'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:26.745384 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b9e9e025-e58f-47a1-b056-678848157320 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd8444f6398e04554a7907a2d3e0eb1cf'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:26.779847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 867/1724] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:26 2022] DELETE /identity/v3/users/d8444f6398e04554a7907a2d3e0eb1cf => generated 0 bytes in 450 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:26.782618 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c30d56-b1d3-47ae-aad0-500c4fc468db None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:26.785121 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c30d56-b1d3-47ae-aad0-500c4fc468db None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:26.785555 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c30d56-b1d3-47ae-aad0-500c4fc468db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:26.850166 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-20c30d56-b1d3-47ae-aad0-500c4fc468db None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:26.851414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-20c30d56-b1d3-47ae-aad0-500c4fc468db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:27.007061 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:27.023485 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 858/1725] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:26 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 366 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:27.028787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6488635b-7f0a-42bb-bd7e-9d4b797212dc None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:27.032380 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6488635b-7f0a-42bb-bd7e-9d4b797212dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:27.041432 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 868/1726] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:27.042841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c30d56-b1d3-47ae-aad0-500c4fc468db None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:27.048029 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-20c30d56-b1d3-47ae-aad0-500c4fc468db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:27.057525 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6488635b-7f0a-42bb-bd7e-9d4b797212dc None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:27.069519 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7d112b-1e4e-44af-aed3-b6f872a84056 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:27.070288 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7d112b-1e4e-44af-aed3-b6f872a84056 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:27.070674 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7d112b-1e4e-44af-aed3-b6f872a84056 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:27.071054 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7d112b-1e4e-44af-aed3-b6f872a84056 admin admin] PATH_INFO: `/v3/projects/f4586bb7af2347089f33c5d9388e1eee` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:27.073612 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7d112b-1e4e-44af-aed3-b6f872a84056 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f4586bb7af2347089f33c5d9388e1eee)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:27.076194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7d112b-1e4e-44af-aed3-b6f872a84056 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:27.093283 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c30d56-b1d3-47ae-aad0-500c4fc468db None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:27.100239 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d38998-8cd3-4360-a994-ce05fed13e85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:27.102040 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d38998-8cd3-4360-a994-ce05fed13e85 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:27.102564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d38998-8cd3-4360-a994-ce05fed13e85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:27.103197 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d38998-8cd3-4360-a994-ce05fed13e85 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:27.104361 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d38998-8cd3-4360-a994-ce05fed13e85 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:27.111477 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d38998-8cd3-4360-a994-ce05fed13e85 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:27.174689 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 869/1727] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:27 2022] POST /identity/v3/projects => generated 405 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:27.176612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d38998-8cd3-4360-a994-ce05fed13e85 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:27.178522 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-76d38998-8cd3-4360-a994-ce05fed13e85 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:27.190459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7c7d112b-1e4e-44af-aed3-b6f872a84056 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f4586bb7af2347089f33c5d9388e1eee', 'request_id': 'req-7c7d112b-1e4e-44af-aed3-b6f872a84056'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:27.205090 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7c7d112b-1e4e-44af-aed3-b6f872a84056 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f4586bb7af2347089f33c5d9388e1eee', 'request_id': 'req-7c7d112b-1e4e-44af-aed3-b6f872a84056'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:27.216621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7c7d112b-1e4e-44af-aed3-b6f872a84056 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f4586bb7af2347089f33c5d9388e1eee', 'request_id': 'req-7c7d112b-1e4e-44af-aed3-b6f872a84056'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:27.238544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7c7d112b-1e4e-44af-aed3-b6f872a84056 admin admin] The token cache is being invalidate because project f4586bb7af2347089f33c5d9388e1eee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:27.239086 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7c7d112b-1e4e-44af-aed3-b6f872a84056 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:27.240989 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7c7d112b-1e4e-44af-aed3-b6f872a84056 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:27.244133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 859/1728] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:27 2022] DELETE /identity/v3/projects/f4586bb7af2347089f33c5d9388e1eee => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:27.245672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7d112b-1e4e-44af-aed3-b6f872a84056 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:27.254200 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7c7d112b-1e4e-44af-aed3-b6f872a84056 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:27.294207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:27.295731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d38998-8cd3-4360-a994-ce05fed13e85 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:27.304332 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6f3380-ee68-41b3-9227-7411da7d3a71 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:27.305302 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6f3380-ee68-41b3-9227-7411da7d3a71 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:27.305858 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6f3380-ee68-41b3-9227-7411da7d3a71 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:27.306383 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6f3380-ee68-41b3-9227-7411da7d3a71 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:27.325591 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8a6f3380-ee68-41b3-9227-7411da7d3a71 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:27.411572 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7d112b-1e4e-44af-aed3-b6f872a84056 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:27.544030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9869685-fc57-4c7f-b9ed-4a6a996cf2a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:27.544686 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9869685-fc57-4c7f-b9ed-4a6a996cf2a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:27.544927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9869685-fc57-4c7f-b9ed-4a6a996cf2a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:27.545236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9869685-fc57-4c7f-b9ed-4a6a996cf2a2 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:27.545887 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9869685-fc57-4c7f-b9ed-4a6a996cf2a2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:27.548144 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9869685-fc57-4c7f-b9ed-4a6a996cf2a2 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:27.569757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6f3380-ee68-41b3-9227-7411da7d3a71 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:27.571378 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6f3380-ee68-41b3-9227-7411da7d3a71 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:27.580904 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 860/1729] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:27 2022] POST /identity/v3/users => generated 309 bytes in 336 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:27.583171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9869685-fc57-4c7f-b9ed-4a6a996cf2a2 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:27.591936 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b9869685-fc57-4c7f-b9ed-4a6a996cf2a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:27.597672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 870/1730] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:27.603968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6f3380-ee68-41b3-9227-7411da7d3a71 service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:27.622773 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6f3380-ee68-41b3-9227-7411da7d3a71 service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:27.629539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723e3bc7-02d6-4488-a389-4a0bbae1ccc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:27.630524 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723e3bc7-02d6-4488-a389-4a0bbae1ccc2 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:27.631020 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723e3bc7-02d6-4488-a389-4a0bbae1ccc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:27.631512 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723e3bc7-02d6-4488-a389-4a0bbae1ccc2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:27.632417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723e3bc7-02d6-4488-a389-4a0bbae1ccc2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:27.634696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9869685-fc57-4c7f-b9ed-4a6a996cf2a2 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:27.635097 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723e3bc7-02d6-4488-a389-4a0bbae1ccc2 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:27.647999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa07e85-9073-4ded-a521-a939ad4d74c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:27.648709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa07e85-9073-4ded-a521-a939ad4d74c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:27.649016 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa07e85-9073-4ded-a521-a939ad4d74c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:27.652941 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa07e85-9073-4ded-a521-a939ad4d74c8 admin admin] PATH_INFO: `/v3/users/b71e2b5f9aa348c6baa2f7def68ad306` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:27.656444 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa07e85-9073-4ded-a521-a939ad4d74c8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b71e2b5f9aa348c6baa2f7def68ad306)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:27.658443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa07e85-9073-4ded-a521-a939ad4d74c8 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:27.659347 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 871/1731] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:27 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:27.684307 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723e3bc7-02d6-4488-a389-4a0bbae1ccc2 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:27.705998 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723e3bc7-02d6-4488-a389-4a0bbae1ccc2 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:27.712034 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bb859c-92e3-44cd-bd02-70104849c3f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:27.717374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bb859c-92e3-44cd-bd02-70104849c3f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:27.717738 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bb859c-92e3-44cd-bd02-70104849c3f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:27.718128 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bb859c-92e3-44cd-bd02-70104849c3f2 admin admin] PATH_INFO: `/v3/projects/0e152cdcc7d042de92d06a01e0b3ec21/users/42ba03515f124c1094899574cacf86a7/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:27.749909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bb859c-92e3-44cd-bd02-70104849c3f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e152cdcc7d042de92d06a01e0b3ec21, user_id=42ba03515f124c1094899574cacf86a7, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:27.754724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bb859c-92e3-44cd-bd02-70104849c3f2 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:27.789411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 872/1732] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:27 2022] PUT /identity/v3/projects/0e152cdcc7d042de92d06a01e0b3ec21/users/42ba03515f124c1094899574cacf86a7/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:27.791254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bb859c-92e3-44cd-bd02-70104849c3f2 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:27.794879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-69bb859c-92e3-44cd-bd02-70104849c3f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:27.821879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5fa07e85-9073-4ded-a521-a939ad4d74c8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b71e2b5f9aa348c6baa2f7def68ad306'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:27.894969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5fa07e85-9073-4ded-a521-a939ad4d74c8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b71e2b5f9aa348c6baa2f7def68ad306'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:27.914494 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5fa07e85-9073-4ded-a521-a939ad4d74c8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b71e2b5f9aa348c6baa2f7def68ad306'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:27.917731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:27.933523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5fa07e85-9073-4ded-a521-a939ad4d74c8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b71e2b5f9aa348c6baa2f7def68ad306'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:27.934222 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5fa07e85-9073-4ded-a521-a939ad4d74c8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b71e2b5f9aa348c6baa2f7def68ad306'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:27.964253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bb859c-92e3-44cd-bd02-70104849c3f2 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:27.980193 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 861/1733] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:27 2022] DELETE /identity/v3/users/b71e2b5f9aa348c6baa2f7def68ad306 => generated 0 bytes in 398 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:27.983143 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa07e85-9073-4ded-a521-a939ad4d74c8 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:27.993328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5fa07e85-9073-4ded-a521-a939ad4d74c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:28.114249 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:28.184110 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e194b1-e36a-423b-99d9-4458c59451cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:28.186190 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e194b1-e36a-423b-99d9-4458c59451cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:28.186642 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e194b1-e36a-423b-99d9-4458c59451cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:28.186924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e194b1-e36a-423b-99d9-4458c59451cb admin admin] PATH_INFO: `/v3/users/aafd1566d1ed4174abe8de556bf8d001` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:28.190277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e194b1-e36a-423b-99d9-4458c59451cb admin admin] RBAC: Authorizing `identity:delete_user(user_id=aafd1566d1ed4174abe8de556bf8d001)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:28.193828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e194b1-e36a-423b-99d9-4458c59451cb admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:28.232455 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa07e85-9073-4ded-a521-a939ad4d74c8 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:28.259879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac01a5f2-2fe8-4c12-9cb7-b53653ec4c74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:28.273717 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac01a5f2-2fe8-4c12-9cb7-b53653ec4c74 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:28.274026 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac01a5f2-2fe8-4c12-9cb7-b53653ec4c74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:28.274327 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac01a5f2-2fe8-4c12-9cb7-b53653ec4c74 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:28.275074 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac01a5f2-2fe8-4c12-9cb7-b53653ec4c74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:28.318819 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-61e194b1-e36a-423b-99d9-4458c59451cb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aafd1566d1ed4174abe8de556bf8d001'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:28.332230 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac01a5f2-2fe8-4c12-9cb7-b53653ec4c74 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:28.334996 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-61e194b1-e36a-423b-99d9-4458c59451cb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aafd1566d1ed4174abe8de556bf8d001'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:28.335769 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-61e194b1-e36a-423b-99d9-4458c59451cb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aafd1566d1ed4174abe8de556bf8d001'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:28.380267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-61e194b1-e36a-423b-99d9-4458c59451cb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aafd1566d1ed4174abe8de556bf8d001'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:28.385871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-61e194b1-e36a-423b-99d9-4458c59451cb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aafd1566d1ed4174abe8de556bf8d001'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:28.392099 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 862/1734] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:27 2022] GET /identity/v3/roles => generated 1420 bytes in 405 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:28.398248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac01a5f2-2fe8-4c12-9cb7-b53653ec4c74 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:28.404496 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ac01a5f2-2fe8-4c12-9cb7-b53653ec4c74 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:28.431554 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 873/1735] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:27 2022] DELETE /identity/v3/users/aafd1566d1ed4174abe8de556bf8d001 => generated 0 bytes in 641 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:28.436037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e414649-4642-4008-8ed0-8128d381f38c None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:28.436460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e414649-4642-4008-8ed0-8128d381f38c None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:28.436848 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e414649-4642-4008-8ed0-8128d381f38c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:28.457997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac01a5f2-2fe8-4c12-9cb7-b53653ec4c74 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:28.485684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-6e414649-4642-4008-8ed0-8128d381f38c None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:28.487296 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-6e414649-4642-4008-8ed0-8128d381f38c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:28.622150 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcafe68d-8ef8-4877-bba6-bd76303a39ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:28.625831 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcafe68d-8ef8-4877-bba6-bd76303a39ff admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:28.627852 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcafe68d-8ef8-4877-bba6-bd76303a39ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:28.628148 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcafe68d-8ef8-4877-bba6-bd76303a39ff admin admin] PATH_INFO: `/v3/projects/0e152cdcc7d042de92d06a01e0b3ec21/users/42ba03515f124c1094899574cacf86a7/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:28.638623 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcafe68d-8ef8-4877-bba6-bd76303a39ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e152cdcc7d042de92d06a01e0b3ec21, user_id=42ba03515f124c1094899574cacf86a7, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:28.648339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcafe68d-8ef8-4877-bba6-bd76303a39ff admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:28.670135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 874/1736] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:28.684042 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 863/1737] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:28 2022] PUT /identity/v3/projects/0e152cdcc7d042de92d06a01e0b3ec21/users/42ba03515f124c1094899574cacf86a7/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:28.754620 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e414649-4642-4008-8ed0-8128d381f38c None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:28.756428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-6e414649-4642-4008-8ed0-8128d381f38c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:28.792001 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:28.860781 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e414649-4642-4008-8ed0-8128d381f38c None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:28.974166 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa678644-57fc-4e6b-824f-f8fcc63760d1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:28.978009 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa678644-57fc-4e6b-824f-f8fcc63760d1 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:28.978756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa678644-57fc-4e6b-824f-f8fcc63760d1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:28.979148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa678644-57fc-4e6b-824f-f8fcc63760d1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:29.099456 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcafe68d-8ef8-4877-bba6-bd76303a39ff admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:29.110831 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bcafe68d-8ef8-4877-bba6-bd76303a39ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:29.252163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa678644-57fc-4e6b-824f-f8fcc63760d1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:29.254940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa678644-57fc-4e6b-824f-f8fcc63760d1 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:29.294471 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 875/1738] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 541 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:29.385652 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa678644-57fc-4e6b-824f-f8fcc63760d1 service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:29.389004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-aa678644-57fc-4e6b-824f-f8fcc63760d1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:29.445249 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa678644-57fc-4e6b-824f-f8fcc63760d1 service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:29.473903 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8a745c-7255-4908-929e-78417e541779 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:29.474573 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8a745c-7255-4908-929e-78417e541779 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:29.474860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8a745c-7255-4908-929e-78417e541779 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:29.475195 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8a745c-7255-4908-929e-78417e541779 admin admin] PATH_INFO: `/v3/projects/44cec317ac97406d8909fc153756808b` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:29.477768 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8a745c-7255-4908-929e-78417e541779 admin admin] RBAC: Authorizing `identity:delete_project(project_id=44cec317ac97406d8909fc153756808b)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:29.484508 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8a745c-7255-4908-929e-78417e541779 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:29.522002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcafe68d-8ef8-4877-bba6-bd76303a39ff admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:29.536410 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2ee4a9-865e-43ff-b36a-e4026f95cfbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:29.537345 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2ee4a9-865e-43ff-b36a-e4026f95cfbf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:29.544365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2ee4a9-865e-43ff-b36a-e4026f95cfbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:29.544993 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2ee4a9-865e-43ff-b36a-e4026f95cfbf admin admin] PATH_INFO: `/v3/projects/73ccb2772b4a42358fd022daab0c5356` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:29.549267 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2ee4a9-865e-43ff-b36a-e4026f95cfbf admin admin] RBAC: Authorizing `identity:delete_project(project_id=73ccb2772b4a42358fd022daab0c5356)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:29.558615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2ee4a9-865e-43ff-b36a-e4026f95cfbf admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:29.682298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-df8a745c-7255-4908-929e-78417e541779 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '44cec317ac97406d8909fc153756808b', 'request_id': 'req-df8a745c-7255-4908-929e-78417e541779'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:29.698380 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-df8a745c-7255-4908-929e-78417e541779 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '44cec317ac97406d8909fc153756808b', 'request_id': 'req-df8a745c-7255-4908-929e-78417e541779'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:29.720591 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-df8a745c-7255-4908-929e-78417e541779 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '44cec317ac97406d8909fc153756808b', 'request_id': 'req-df8a745c-7255-4908-929e-78417e541779'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:29.737868 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-bc2ee4a9-865e-43ff-b36a-e4026f95cfbf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '73ccb2772b4a42358fd022daab0c5356', 'request_id': 'req-bc2ee4a9-865e-43ff-b36a-e4026f95cfbf'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:29.757784 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-bc2ee4a9-865e-43ff-b36a-e4026f95cfbf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '73ccb2772b4a42358fd022daab0c5356', 'request_id': 'req-bc2ee4a9-865e-43ff-b36a-e4026f95cfbf'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:29.765915 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:29.767140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-df8a745c-7255-4908-929e-78417e541779 admin admin] The token cache is being invalidate because project 44cec317ac97406d8909fc153756808b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:29.767524 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-df8a745c-7255-4908-929e-78417e541779 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:29.771262 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-df8a745c-7255-4908-929e-78417e541779 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:29.775066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 876/1739] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:29 2022] DELETE /identity/v3/projects/44cec317ac97406d8909fc153756808b => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:29.778340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1910c0b4-b7f0-4647-a1e2-495fef873f4b None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:29.780631 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1910c0b4-b7f0-4647-a1e2-495fef873f4b None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:29.781065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1910c0b4-b7f0-4647-a1e2-495fef873f4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:29.787168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-bc2ee4a9-865e-43ff-b36a-e4026f95cfbf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '73ccb2772b4a42358fd022daab0c5356', 'request_id': 'req-bc2ee4a9-865e-43ff-b36a-e4026f95cfbf'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:29.799370 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:29.832718 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-bc2ee4a9-865e-43ff-b36a-e4026f95cfbf admin admin] The token cache is being invalidate because project 73ccb2772b4a42358fd022daab0c5356 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:29.834672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-1910c0b4-b7f0-4647-a1e2-495fef873f4b None None] MFA Rules not processed for user `298b05a8580946f4ab2ed66f0b9a3c2b`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:29.835112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-bc2ee4a9-865e-43ff-b36a-e4026f95cfbf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:29.836095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-1910c0b4-b7f0-4647-a1e2-495fef873f4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:29.839398 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-bc2ee4a9-865e-43ff-b36a-e4026f95cfbf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:29.844671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 864/1740] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:29 2022] DELETE /identity/v3/projects/73ccb2772b4a42358fd022daab0c5356 => generated 0 bytes in 745 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:29.849865 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de21c78a-0bfe-4d75-93a6-1030f6fe4d35 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:29.851420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de21c78a-0bfe-4d75-93a6-1030f6fe4d35 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:29.854916 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de21c78a-0bfe-4d75-93a6-1030f6fe4d35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:29.966889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-de21c78a-0bfe-4d75-93a6-1030f6fe4d35 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:29.976350 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-de21c78a-0bfe-4d75-93a6-1030f6fe4d35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:29.996363 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 877/1741] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:29 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:29.997837 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12d755b-00c7-4e50-af93-2b96badbb1a5 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:29.998132 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12d755b-00c7-4e50-af93-2b96badbb1a5 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:29.998405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12d755b-00c7-4e50-af93-2b96badbb1a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:30.048755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-f12d755b-00c7-4e50-af93-2b96badbb1a5 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:30.050186 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f12d755b-00c7-4e50-af93-2b96badbb1a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:30.187881 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 878/1742] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:30.194128 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e287a3-7100-4b0b-a1c4-86334ef7112a None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:30.194793 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e287a3-7100-4b0b-a1c4-86334ef7112a None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:30.195427 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e287a3-7100-4b0b-a1c4-86334ef7112a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:30.240187 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 865/1743] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 398 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:30.243356 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51035f51-d46d-4b4b-8e2c-75e3cb1d92ee None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:30.246904 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51035f51-d46d-4b4b-8e2c-75e3cb1d92ee None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:30.248203 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51035f51-d46d-4b4b-8e2c-75e3cb1d92ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:30.266712 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-e0e287a3-7100-4b0b-a1c4-86334ef7112a None None] MFA Rules not processed for user `298b05a8580946f4ab2ed66f0b9a3c2b`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:30.268240 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e0e287a3-7100-4b0b-a1c4-86334ef7112a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:30.280907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 879/1744] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:30 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:30.280907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a6bd39-482e-41f4-b161-18f1c2bbe05e None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:30.280907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a6bd39-482e-41f4-b161-18f1c2bbe05e None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:30.280907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a6bd39-482e-41f4-b161-18f1c2bbe05e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:30.339952 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-d4a6bd39-482e-41f4-b161-18f1c2bbe05e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:30.341385 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d4a6bd39-482e-41f4-b161-18f1c2bbe05e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:30.351785 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 880/1745] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:30.352637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a6bd39-482e-41f4-b161-18f1c2bbe05e None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:30.354618 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d4a6bd39-482e-41f4-b161-18f1c2bbe05e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:30.397134 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-51035f51-d46d-4b4b-8e2c-75e3cb1d92ee None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:30.413798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-51035f51-d46d-4b4b-8e2c-75e3cb1d92ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:30.473960 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 866/1746] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:30.476532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51035f51-d46d-4b4b-8e2c-75e3cb1d92ee None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:30.514138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a6bd39-482e-41f4-b161-18f1c2bbe05e None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:30.516529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51035f51-d46d-4b4b-8e2c-75e3cb1d92ee None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:30.533797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b824fc9-8fe5-47f4-8b5f-d1626818570d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:30.533797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b824fc9-8fe5-47f4-8b5f-d1626818570d service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:30.533797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b824fc9-8fe5-47f4-8b5f-d1626818570d service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:30.533797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b824fc9-8fe5-47f4-8b5f-d1626818570d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:30.541209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379792f2-4d6c-4811-95c8-a5b9e9aabb66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:30.542359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379792f2-4d6c-4811-95c8-a5b9e9aabb66 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:30.542664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379792f2-4d6c-4811-95c8-a5b9e9aabb66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:30.543240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379792f2-4d6c-4811-95c8-a5b9e9aabb66 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:30.544310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379792f2-4d6c-4811-95c8-a5b9e9aabb66 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:30.544627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379792f2-4d6c-4811-95c8-a5b9e9aabb66 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:30.546992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379792f2-4d6c-4811-95c8-a5b9e9aabb66 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:30.547223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b824fc9-8fe5-47f4-8b5f-d1626818570d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:30.565012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b824fc9-8fe5-47f4-8b5f-d1626818570d service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:30.569340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 867/1747] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:26:30.570568 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379792f2-4d6c-4811-95c8-a5b9e9aabb66 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:30.586112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379792f2-4d6c-4811-95c8-a5b9e9aabb66 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:30.604582 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c497929-efbb-4d32-8adf-467255c6f12e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:30.605463 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c497929-efbb-4d32-8adf-467255c6f12e admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:30.605743 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c497929-efbb-4d32-8adf-467255c6f12e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:30.606048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c497929-efbb-4d32-8adf-467255c6f12e admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:30.606739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c497929-efbb-4d32-8adf-467255c6f12e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:30.607081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c497929-efbb-4d32-8adf-467255c6f12e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:30.607720 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 881/1748] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:30 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:30.608781 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b824fc9-8fe5-47f4-8b5f-d1626818570d service nova] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:30.610177 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c497929-efbb-4d32-8adf-467255c6f12e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:30.629431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 868/1749] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:26:30.633517 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c497929-efbb-4d32-8adf-467255c6f12e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:30.648967 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c497929-efbb-4d32-8adf-467255c6f12e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:30.656523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b902fe12-6ddb-4117-ac01-35571317c956 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:30.657329 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b824fc9-8fe5-47f4-8b5f-d1626818570d service nova] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:30.657461 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b902fe12-6ddb-4117-ac01-35571317c956 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:30.657781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b902fe12-6ddb-4117-ac01-35571317c956 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:30.658245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b902fe12-6ddb-4117-ac01-35571317c956 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:30.658984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b902fe12-6ddb-4117-ac01-35571317c956 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:30.661880 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b902fe12-6ddb-4117-ac01-35571317c956 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:30.677988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870da3b0-317c-4829-9173-c4d5ea3045ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:30.678821 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870da3b0-317c-4829-9173-c4d5ea3045ee admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:30.679252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870da3b0-317c-4829-9173-c4d5ea3045ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:30.679361 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870da3b0-317c-4829-9173-c4d5ea3045ee admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:30.680014 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870da3b0-317c-4829-9173-c4d5ea3045ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:30.682209 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870da3b0-317c-4829-9173-c4d5ea3045ee admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:30.703714 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:30.764962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 882/1750] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:30 2022] POST /identity/v3/projects => generated 377 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:30.768856 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870da3b0-317c-4829-9173-c4d5ea3045ee admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:30.824968 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 869/1751] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:30 2022] POST /identity/v3/projects => generated 371 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:30.830906 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b902fe12-6ddb-4117-ac01-35571317c956 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:30.971282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870da3b0-317c-4829-9173-c4d5ea3045ee admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:31.100317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b902fe12-6ddb-4117-ac01-35571317c956 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:31.124537 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a295a94-9e1b-42f7-bb93-13116be8a01e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:31.127967 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a295a94-9e1b-42f7-bb93-13116be8a01e admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:31.128289 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a295a94-9e1b-42f7-bb93-13116be8a01e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:31.128857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a295a94-9e1b-42f7-bb93-13116be8a01e admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:31.130217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a295a94-9e1b-42f7-bb93-13116be8a01e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:31.140175 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a295a94-9e1b-42f7-bb93-13116be8a01e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:31.171027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40afc582-668f-44bc-8419-b9bd591f64cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:31.171701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40afc582-668f-44bc-8419-b9bd591f64cf admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:31.171983 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40afc582-668f-44bc-8419-b9bd591f64cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:31.172255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40afc582-668f-44bc-8419-b9bd591f64cf admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:31.173462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40afc582-668f-44bc-8419-b9bd591f64cf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:31.176546 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40afc582-668f-44bc-8419-b9bd591f64cf admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:31.227171 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 883/1752] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:30 2022] POST /identity/v3/users => generated 289 bytes in 458 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:31.228911 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b99620-72c2-4bda-b377-9aec58c00fc8 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:31.229369 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b99620-72c2-4bda-b377-9aec58c00fc8 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:31.229934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b99620-72c2-4bda-b377-9aec58c00fc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:31.278308 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 870/1753] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:30 2022] POST /identity/v3/users => generated 286 bytes in 438 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:31.278308 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a295a94-9e1b-42f7-bb93-13116be8a01e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:31.315427 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a295a94-9e1b-42f7-bb93-13116be8a01e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:31.320455 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-66b99620-72c2-4bda-b377-9aec58c00fc8 None None] MFA Rules not processed for user `00aeab772672403984bdcb8baaa1520e`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:31.323172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-66b99620-72c2-4bda-b377-9aec58c00fc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:31.332810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70065767-0c4e-400f-9e9c-ecd227c126eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:31.335556 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70065767-0c4e-400f-9e9c-ecd227c126eb admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:31.337578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70065767-0c4e-400f-9e9c-ecd227c126eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:31.338030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70065767-0c4e-400f-9e9c-ecd227c126eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:31.338943 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70065767-0c4e-400f-9e9c-ecd227c126eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:31.345748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70065767-0c4e-400f-9e9c-ecd227c126eb admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:31.395595 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 871/1754] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:31 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:31.396370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70065767-0c4e-400f-9e9c-ecd227c126eb admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:31.433488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70065767-0c4e-400f-9e9c-ecd227c126eb admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:31.446924 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bce915-7443-46d8-be13-00cb7e261f27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:31.447563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bce915-7443-46d8-be13-00cb7e261f27 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:31.447843 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bce915-7443-46d8-be13-00cb7e261f27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:31.448597 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bce915-7443-46d8-be13-00cb7e261f27 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:31.449469 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bce915-7443-46d8-be13-00cb7e261f27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:31.452479 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bce915-7443-46d8-be13-00cb7e261f27 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:31.473296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:INSERT=5,keystone:DELETE=1,keystone:SELECT=94 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:31.496649 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 872/1755] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:31 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:31.497641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bce915-7443-46d8-be13-00cb7e261f27 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:31.509172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:31.542564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 884/1756] 10.222.0.85 () {58 vars in 1009 bytes} [Tue Aug 30 04:26:31 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 315 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:31.543538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b99620-72c2-4bda-b377-9aec58c00fc8 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:31.548400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bce915-7443-46d8-be13-00cb7e261f27 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:31.555374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ecf4fa-f3b5-4a70-9802-e74a96d69567 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:31.555995 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ecf4fa-f3b5-4a70-9802-e74a96d69567 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:31.556275 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ecf4fa-f3b5-4a70-9802-e74a96d69567 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:31.558258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ecf4fa-f3b5-4a70-9802-e74a96d69567 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:31.561206 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ecf4fa-f3b5-4a70-9802-e74a96d69567 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:31.563866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ecf4fa-f3b5-4a70-9802-e74a96d69567 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:31.597620 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b99620-72c2-4bda-b377-9aec58c00fc8 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:31.616093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0923c45-b1fb-4ee9-83ce-d888f09023ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:31.619145 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0923c45-b1fb-4ee9-83ce-d888f09023ae admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:31.619511 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0923c45-b1fb-4ee9-83ce-d888f09023ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:31.622648 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0923c45-b1fb-4ee9-83ce-d888f09023ae admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:31.623494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0923c45-b1fb-4ee9-83ce-d888f09023ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:31.625505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 873/1757] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:31 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:31.631377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0923c45-b1fb-4ee9-83ce-d888f09023ae admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:31.646046 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ecf4fa-f3b5-4a70-9802-e74a96d69567 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:31.690115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ecf4fa-f3b5-4a70-9802-e74a96d69567 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:31.691285 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 885/1758] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:31 2022] GET /identity/v3/roles => generated 1420 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:31.700437 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0923c45-b1fb-4ee9-83ce-d888f09023ae admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:31.705356 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c0923c45-b1fb-4ee9-83ce-d888f09023ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:31.711421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02b9f38-9f88-45e2-b343-f5bdbc0df044 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:31.716107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02b9f38-9f88-45e2-b343-f5bdbc0df044 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:31.718783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02b9f38-9f88-45e2-b343-f5bdbc0df044 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:31.719036 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02b9f38-9f88-45e2-b343-f5bdbc0df044 admin admin] PATH_INFO: `/v3/projects/2ef1fad6e4f7424abdb598e740902018/users/9dfedf5a775944b1b8b665e5654efe8b/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:31.744227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0923c45-b1fb-4ee9-83ce-d888f09023ae admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:31.755263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b095be55-9ced-43a0-81a2-72d538e13958 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:31.755921 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b095be55-9ced-43a0-81a2-72d538e13958 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:31.756202 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b095be55-9ced-43a0-81a2-72d538e13958 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:31.756479 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b095be55-9ced-43a0-81a2-72d538e13958 admin admin] PATH_INFO: `/v3/projects/d1de048add3b4fc79401df1d0952f67a` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:31.758667 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b095be55-9ced-43a0-81a2-72d538e13958 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d1de048add3b4fc79401df1d0952f67a)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:31.760624 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b095be55-9ced-43a0-81a2-72d538e13958 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:31.816401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02b9f38-9f88-45e2-b343-f5bdbc0df044 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ef1fad6e4f7424abdb598e740902018, user_id=9dfedf5a775944b1b8b665e5654efe8b, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:31.827586 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02b9f38-9f88-45e2-b343-f5bdbc0df044 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:31.922169 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 874/1759] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:31 2022] PUT /identity/v3/projects/2ef1fad6e4f7424abdb598e740902018/users/9dfedf5a775944b1b8b665e5654efe8b/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:31.923569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02b9f38-9f88-45e2-b343-f5bdbc0df044 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:31.929086 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b095be55-9ced-43a0-81a2-72d538e13958 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd1de048add3b4fc79401df1d0952f67a', 'request_id': 'req-b095be55-9ced-43a0-81a2-72d538e13958'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:31.939473 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b095be55-9ced-43a0-81a2-72d538e13958 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd1de048add3b4fc79401df1d0952f67a', 'request_id': 'req-b095be55-9ced-43a0-81a2-72d538e13958'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:31.962976 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b095be55-9ced-43a0-81a2-72d538e13958 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd1de048add3b4fc79401df1d0952f67a', 'request_id': 'req-b095be55-9ced-43a0-81a2-72d538e13958'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:31.997211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b095be55-9ced-43a0-81a2-72d538e13958 admin admin] The token cache is being invalidate because project d1de048add3b4fc79401df1d0952f67a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:31.997465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b095be55-9ced-43a0-81a2-72d538e13958 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:32.000825 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b095be55-9ced-43a0-81a2-72d538e13958 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:32.005258 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 886/1760] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:31 2022] DELETE /identity/v3/projects/d1de048add3b4fc79401df1d0952f67a => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:32.006278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b095be55-9ced-43a0-81a2-72d538e13958 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:32.012860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b095be55-9ced-43a0-81a2-72d538e13958 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:32.121113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02b9f38-9f88-45e2-b343-f5bdbc0df044 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:32.264619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7360ee33-81f0-4e72-a2c4-6cd9a8f8cdb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:32.265454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7360ee33-81f0-4e72-a2c4-6cd9a8f8cdb5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:32.266891 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7360ee33-81f0-4e72-a2c4-6cd9a8f8cdb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:32.267299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7360ee33-81f0-4e72-a2c4-6cd9a8f8cdb5 admin admin] PATH_INFO: `/v3/projects/aa4e296e79654771af4e3b6bb19d402b/users/f6ff88ebcb8440e99539b0a2cda8e152/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:32.267987 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b095be55-9ced-43a0-81a2-72d538e13958 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:32.293180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad63c1a-3d1e-402e-aba6-8e8b20d0a726 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cb539ef6a69406baa70f660e1a815b4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:32.293820 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad63c1a-3d1e-402e-aba6-8e8b20d0a726 service swift] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:32.294103 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad63c1a-3d1e-402e-aba6-8e8b20d0a726 service swift] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:32.294385 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad63c1a-3d1e-402e-aba6-8e8b20d0a726 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:32.302516 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-1ad63c1a-3d1e-402e-aba6-8e8b20d0a726 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:32.306191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7360ee33-81f0-4e72-a2c4-6cd9a8f8cdb5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa4e296e79654771af4e3b6bb19d402b, user_id=f6ff88ebcb8440e99539b0a2cda8e152, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:32.315203 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7360ee33-81f0-4e72-a2c4-6cd9a8f8cdb5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:32.346757 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 875/1761] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:31 2022] PUT /identity/v3/projects/aa4e296e79654771af4e3b6bb19d402b/users/f6ff88ebcb8440e99539b0a2cda8e152/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 424 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:32.349945 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03431297-cbea-4a0a-adc3-be5906782081 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:32.351724 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03431297-cbea-4a0a-adc3-be5906782081 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:32.352264 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03431297-cbea-4a0a-adc3-be5906782081 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:32.362130 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 876/1762] 10.222.0.85 () {54 vars in 912 bytes} [Tue Aug 30 04:26:32 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:26:32.365249 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbcda06-1229-46fc-a85a-530adbcfecc0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:32.365561 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbcda06-1229-46fc-a85a-530adbcfecc0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:32.366807 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbcda06-1229-46fc-a85a-530adbcfecc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:32.478381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4cbcda06-1229-46fc-a85a-530adbcfecc0 None None] MFA Rules not processed for user `9dfedf5a775944b1b8b665e5654efe8b`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:32.481452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4cbcda06-1229-46fc-a85a-530adbcfecc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:32.481600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:32.522169 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad63c1a-3d1e-402e-aba6-8e8b20d0a726 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:32.523819 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad63c1a-3d1e-402e-aba6-8e8b20d0a726 service swift] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:32.552890 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 887/1763] 10.222.0.85 () {60 vars in 1384 bytes} [Tue Aug 30 04:26:32 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 547 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 04:26:32.553811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad63c1a-3d1e-402e-aba6-8e8b20d0a726 service swift] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:32.555203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-1ad63c1a-3d1e-402e-aba6-8e8b20d0a726 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:32.584757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad63c1a-3d1e-402e-aba6-8e8b20d0a726 service swift] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:32.673927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:32.727122 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 877/1764] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:32 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 364 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:32.730035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55938c88-b4ad-4216-aa48-84cf9aaa821f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:32.732879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55938c88-b4ad-4216-aa48-84cf9aaa821f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:32.733310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55938c88-b4ad-4216-aa48-84cf9aaa821f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:32.781861 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-55938c88-b4ad-4216-aa48-84cf9aaa821f None None] MFA Rules not processed for user `8ef7a83a87d44fc0a5647cb60057c036`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:32.792070 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-55938c88-b4ad-4216-aa48-84cf9aaa821f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:32.798808 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdba40c-307b-4141-95f0-a93312c80b23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:32.799719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdba40c-307b-4141-95f0-a93312c80b23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:32.800195 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdba40c-307b-4141-95f0-a93312c80b23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:32.800620 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdba40c-307b-4141-95f0-a93312c80b23 admin admin] PATH_INFO: `/v3/users/f806c5ab9e804959a93e38eb1c1e0159` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:32.811989 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdba40c-307b-4141-95f0-a93312c80b23 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f806c5ab9e804959a93e38eb1c1e0159)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:32.814517 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdba40c-307b-4141-95f0-a93312c80b23 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:32.931156 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-efdba40c-307b-4141-95f0-a93312c80b23 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f806c5ab9e804959a93e38eb1c1e0159'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:32.947235 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-efdba40c-307b-4141-95f0-a93312c80b23 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f806c5ab9e804959a93e38eb1c1e0159'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:32.948306 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-efdba40c-307b-4141-95f0-a93312c80b23 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f806c5ab9e804959a93e38eb1c1e0159'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:32.956885 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 878/1765] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:32 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:32.959145 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bb3fc0-04ec-43de-a31c-735c9158a2ef None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:32.959145 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bb3fc0-04ec-43de-a31c-735c9158a2ef None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:32.960897 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bb3fc0-04ec-43de-a31c-735c9158a2ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:32.997854 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-efdba40c-307b-4141-95f0-a93312c80b23 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f806c5ab9e804959a93e38eb1c1e0159'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:32.998779 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-efdba40c-307b-4141-95f0-a93312c80b23 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f806c5ab9e804959a93e38eb1c1e0159'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:33.028150 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-32bb3fc0-04ec-43de-a31c-735c9158a2ef None None] MFA Rules not processed for user `42ba03515f124c1094899574cacf86a7`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:33.029608 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-32bb3fc0-04ec-43de-a31c-735c9158a2ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:33.076871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 888/1766] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:32 2022] DELETE /identity/v3/users/f806c5ab9e804959a93e38eb1c1e0159 => generated 0 bytes in 523 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:33.094444 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da27f0f-6d26-4946-ab46-e3c2f10ca298 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:33.094834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da27f0f-6d26-4946-ab46-e3c2f10ca298 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:33.095016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da27f0f-6d26-4946-ab46-e3c2f10ca298 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:33.217340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-5da27f0f-6d26-4946-ab46-e3c2f10ca298 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:33.226617 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-5da27f0f-6d26-4946-ab46-e3c2f10ca298 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:33.264441 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 879/1767] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:32 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:33.271263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1ca39a-8c0b-428e-9fe2-8b6e7eac982e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:33.272204 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1ca39a-8c0b-428e-9fe2-8b6e7eac982e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:33.274631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1ca39a-8c0b-428e-9fe2-8b6e7eac982e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:33.331857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-db1ca39a-8c0b-428e-9fe2-8b6e7eac982e None None] MFA Rules not processed for user `5c2edd77896a4534888d6758ec0ee0bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:33.345880 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-db1ca39a-8c0b-428e-9fe2-8b6e7eac982e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:33.452998 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 889/1768] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:33.458583 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da27f0f-6d26-4946-ab46-e3c2f10ca298 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:33.462205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-5da27f0f-6d26-4946-ab46-e3c2f10ca298 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:33.491785 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da27f0f-6d26-4946-ab46-e3c2f10ca298 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:33.508862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706b6356-9965-427a-adc6-de4f2981afe4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:33.509951 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706b6356-9965-427a-adc6-de4f2981afe4 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:33.510536 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706b6356-9965-427a-adc6-de4f2981afe4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:33.511347 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706b6356-9965-427a-adc6-de4f2981afe4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:33.512481 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706b6356-9965-427a-adc6-de4f2981afe4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:33.516124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706b6356-9965-427a-adc6-de4f2981afe4 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:33.525452 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:33.587055 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 880/1769] 10.222.0.85 () {58 vars in 1009 bytes} [Tue Aug 30 04:26:33 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 322 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:33.621506 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 890/1770] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:33 2022] POST /identity/v3/projects => generated 377 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:33.624764 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1ca39a-8c0b-428e-9fe2-8b6e7eac982e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:33.653695 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1ca39a-8c0b-428e-9fe2-8b6e7eac982e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:33.687934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706b6356-9965-427a-adc6-de4f2981afe4 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:33.690038 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-706b6356-9965-427a-adc6-de4f2981afe4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:33.776541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:33.913874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706b6356-9965-427a-adc6-de4f2981afe4 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:33.914763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d23e225-b434-4ab9-974e-67a70c04a064 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:33.915512 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d23e225-b434-4ab9-974e-67a70c04a064 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:33.915992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d23e225-b434-4ab9-974e-67a70c04a064 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:33.916455 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d23e225-b434-4ab9-974e-67a70c04a064 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:33.917306 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d23e225-b434-4ab9-974e-67a70c04a064 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:33.919855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d23e225-b434-4ab9-974e-67a70c04a064 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:33.926027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b5d6a9-636d-4b3c-8717-a81fa44ea732 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:33.926711 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b5d6a9-636d-4b3c-8717-a81fa44ea732 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:33.926993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b5d6a9-636d-4b3c-8717-a81fa44ea732 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:33.927265 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b5d6a9-636d-4b3c-8717-a81fa44ea732 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:33.969779 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b5d6a9-636d-4b3c-8717-a81fa44ea732 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:33.972893 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b5d6a9-636d-4b3c-8717-a81fa44ea732 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:33.991656 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 881/1771] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:33 2022] POST /identity/v3/users => generated 295 bytes in 367 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:33.994999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d23e225-b434-4ab9-974e-67a70c04a064 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:34.023447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 891/1772] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:26:34.026998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d23e225-b434-4ab9-974e-67a70c04a064 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:34.028954 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2eb3207-1896-4487-89d3-b8172a8fbe42 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:34.029979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2eb3207-1896-4487-89d3-b8172a8fbe42 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:34.031985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2eb3207-1896-4487-89d3-b8172a8fbe42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:34.045999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde2d617-0570-44e1-9bed-f375eac3aeec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:34.055270 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde2d617-0570-44e1-9bed-f375eac3aeec service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:34.055656 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde2d617-0570-44e1-9bed-f375eac3aeec service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:34.056045 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde2d617-0570-44e1-9bed-f375eac3aeec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:34.099784 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde2d617-0570-44e1-9bed-f375eac3aeec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:34.101470 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde2d617-0570-44e1-9bed-f375eac3aeec service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:34.117352 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-b2eb3207-1896-4487-89d3-b8172a8fbe42 None None] MFA Rules not processed for user `5c2edd77896a4534888d6758ec0ee0bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:34.119024 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b2eb3207-1896-4487-89d3-b8172a8fbe42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:34.124873 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 882/1773] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:33 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:34.125644 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde2d617-0570-44e1-9bed-f375eac3aeec service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:34.148567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde2d617-0570-44e1-9bed-f375eac3aeec service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:34.150864 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 892/1774] 10.222.0.71 () {58 vars in 1015 bytes} [Tue Aug 30 04:26:34 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:34.164403 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2729cd9e-330d-4045-8c94-654e113bf466 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:34.165427 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2729cd9e-330d-4045-8c94-654e113bf466 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:34.165797 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2729cd9e-330d-4045-8c94-654e113bf466 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:34.166141 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2729cd9e-330d-4045-8c94-654e113bf466 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:34.166884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2729cd9e-330d-4045-8c94-654e113bf466 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:34.168688 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2729cd9e-330d-4045-8c94-654e113bf466 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:34.181989 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2eb3207-1896-4487-89d3-b8172a8fbe42 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:34.183765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b2eb3207-1896-4487-89d3-b8172a8fbe42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:34.195279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 883/1775] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:34 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:34.195528 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2729cd9e-330d-4045-8c94-654e113bf466 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:34.209728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2eb3207-1896-4487-89d3-b8172a8fbe42 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:34.221193 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfbe461-0eeb-487c-8d1c-5ff0212d4a50 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:34.222098 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfbe461-0eeb-487c-8d1c-5ff0212d4a50 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:34.223668 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfbe461-0eeb-487c-8d1c-5ff0212d4a50 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:34.223804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfbe461-0eeb-487c-8d1c-5ff0212d4a50 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:34.227515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2729cd9e-330d-4045-8c94-654e113bf466 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:34.243396 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9929a44f-e274-4e64-abe6-766ddfbabeed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:34.244151 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9929a44f-e274-4e64-abe6-766ddfbabeed admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:34.244513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfbe461-0eeb-487c-8d1c-5ff0212d4a50 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:34.244631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9929a44f-e274-4e64-abe6-766ddfbabeed admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:34.245150 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9929a44f-e274-4e64-abe6-766ddfbabeed admin admin] PATH_INFO: `/v3/projects/620391fe5ce24837a102c150548c9c82/users/31f19605128c4df3b9d80d6e0ae7096d/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:34.248325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfbe461-0eeb-487c-8d1c-5ff0212d4a50 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:34.284071 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 893/1776] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:34 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:26:34.285510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9929a44f-e274-4e64-abe6-766ddfbabeed admin admin] RBAC: Authorizing `identity:create_grant(project_id=620391fe5ce24837a102c150548c9c82, user_id=31f19605128c4df3b9d80d6e0ae7096d, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:34.288525 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9929a44f-e274-4e64-abe6-766ddfbabeed admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:34.319823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 884/1777] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:34 2022] PUT /identity/v3/projects/620391fe5ce24837a102c150548c9c82/users/31f19605128c4df3b9d80d6e0ae7096d/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:34.320843 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfbe461-0eeb-487c-8d1c-5ff0212d4a50 service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:34.345191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfbe461-0eeb-487c-8d1c-5ff0212d4a50 service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:34.573847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59e7e9e-0c76-4a6a-bd08-5d415be7f66d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:34.574636 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59e7e9e-0c76-4a6a-bd08-5d415be7f66d admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:34.574914 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59e7e9e-0c76-4a6a-bd08-5d415be7f66d admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:34.575187 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59e7e9e-0c76-4a6a-bd08-5d415be7f66d admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:34.575832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59e7e9e-0c76-4a6a-bd08-5d415be7f66d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:34.577713 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59e7e9e-0c76-4a6a-bd08-5d415be7f66d admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:34.613272 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 894/1778] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:34 2022] GET /identity/v3/roles => generated 1420 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:34.617777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9929a44f-e274-4e64-abe6-766ddfbabeed admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:34.632045 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9929a44f-e274-4e64-abe6-766ddfbabeed admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:34.638291 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4ace00-4673-46ab-bf54-36be58adb22d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:34.638978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4ace00-4673-46ab-bf54-36be58adb22d admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:34.639256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4ace00-4673-46ab-bf54-36be58adb22d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:34.640092 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4ace00-4673-46ab-bf54-36be58adb22d admin admin] PATH_INFO: `/v3/projects/620391fe5ce24837a102c150548c9c82/users/31f19605128c4df3b9d80d6e0ae7096d/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:34.642415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4ace00-4673-46ab-bf54-36be58adb22d admin admin] RBAC: Authorizing `identity:create_grant(project_id=620391fe5ce24837a102c150548c9c82, user_id=31f19605128c4df3b9d80d6e0ae7096d, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:34.646797 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4ace00-4673-46ab-bf54-36be58adb22d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:34.679675 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 885/1779] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:34 2022] PUT /identity/v3/projects/620391fe5ce24837a102c150548c9c82/users/31f19605128c4df3b9d80d6e0ae7096d/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:26:34.697922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22bcebd-9bcc-4417-af3d-ce0ac8ae0a6b None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:34.698493 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22bcebd-9bcc-4417-af3d-ce0ac8ae0a6b None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:34.698841 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22bcebd-9bcc-4417-af3d-ce0ac8ae0a6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:34.794369 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-e22bcebd-9bcc-4417-af3d-ce0ac8ae0a6b None None] MFA Rules not processed for user `31f19605128c4df3b9d80d6e0ae7096d`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:34.801244 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e22bcebd-9bcc-4417-af3d-ce0ac8ae0a6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:35.018754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:35.095445 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 895/1780] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:34 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 392 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:35.096616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4ace00-4673-46ab-bf54-36be58adb22d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:35.099744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fc4ace00-4673-46ab-bf54-36be58adb22d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:35.277832 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4ace00-4673-46ab-bf54-36be58adb22d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:35.304111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0568b81e-3372-4147-95e4-be7a06855eb4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:35.305598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0568b81e-3372-4147-95e4-be7a06855eb4 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:35.309574 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0568b81e-3372-4147-95e4-be7a06855eb4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:35.310360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0568b81e-3372-4147-95e4-be7a06855eb4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:35.316116 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0568b81e-3372-4147-95e4-be7a06855eb4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:35.504171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0568b81e-3372-4147-95e4-be7a06855eb4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:35.507441 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0568b81e-3372-4147-95e4-be7a06855eb4 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:35.529400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 886/1781] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:35 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:26:35.767563 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a817064d-419d-47d0-b663-330321b72ad3 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:35.768522 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a817064d-419d-47d0-b663-330321b72ad3 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:35.768998 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a817064d-419d-47d0-b663-330321b72ad3 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:35.771045 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 896/1782] 10.222.0.85 () {54 vars in 921 bytes} [Tue Aug 30 04:26:35 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:26:35.808057 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9a6a1b-55c4-448f-a05e-83e105b1f492 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:35.809095 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9a6a1b-55c4-448f-a05e-83e105b1f492 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:35.809541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9a6a1b-55c4-448f-a05e-83e105b1f492 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:35.843194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:35.862224 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-8b9a6a1b-55c4-448f-a05e-83e105b1f492 None None] MFA Rules not processed for user `359d79cbfcad49beba2b495066824128`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:35.865281 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8b9a6a1b-55c4-448f-a05e-83e105b1f492 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:36.056650 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 887/1783] 10.222.0.85 () {58 vars in 1018 bytes} [Tue Aug 30 04:26:35 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:36.298474 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fbcfa6-3828-4458-bbe3-2516f10f409b None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:36.298777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fbcfa6-3828-4458-bbe3-2516f10f409b None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:36.299054 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fbcfa6-3828-4458-bbe3-2516f10f409b None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:36.300458 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 897/1784] 10.222.0.85 () {54 vars in 961 bytes} [Tue Aug 30 04:26:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:26:36.307813 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589aaa37-18ff-4f3a-9c5d-196334f3f4c3 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:36.308813 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589aaa37-18ff-4f3a-9c5d-196334f3f4c3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:36.309299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589aaa37-18ff-4f3a-9c5d-196334f3f4c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:36.404894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-589aaa37-18ff-4f3a-9c5d-196334f3f4c3 None None] MFA Rules not processed for user `359d79cbfcad49beba2b495066824128`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:36.418718 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-589aaa37-18ff-4f3a-9c5d-196334f3f4c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:36.469001 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fbcfa6-3828-4458-bbe3-2516f10f409b None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:36.469764 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 888/1785] 10.222.0.85 () {58 vars in 1058 bytes} [Tue Aug 30 04:26:36 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:36.508803 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fbcfa6-3828-4458-bbe3-2516f10f409b None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:36.524406 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d4ad72-e5f8-4c9b-b184-eaac1a4a6076 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '359d79cbfcad49beba2b495066824128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:36.526409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d4ad72-e5f8-4c9b-b184-eaac1a4a6076 service placement] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:36.527693 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d4ad72-e5f8-4c9b-b184-eaac1a4a6076 service placement] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:36.529341 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d4ad72-e5f8-4c9b-b184-eaac1a4a6076 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:36.577030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d4ad72-e5f8-4c9b-b184-eaac1a4a6076 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:36.587317 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d4ad72-e5f8-4c9b-b184-eaac1a4a6076 service placement] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:36.659576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 898/1786] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:36 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:37.901289 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589aaa37-18ff-4f3a-9c5d-196334f3f4c3 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:37.903774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-589aaa37-18ff-4f3a-9c5d-196334f3f4c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:38.235870 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589aaa37-18ff-4f3a-9c5d-196334f3f4c3 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:38.270727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0e4f69-3d32-4e1c-96cf-025bb7b9d5d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:38.271470 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0e4f69-3d32-4e1c-96cf-025bb7b9d5d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:38.271667 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0e4f69-3d32-4e1c-96cf-025bb7b9d5d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:38.271937 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0e4f69-3d32-4e1c-96cf-025bb7b9d5d5 admin admin] PATH_INFO: `/v3/projects/9d68adf14afe47df9c512149e4d01002` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:38.276988 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0e4f69-3d32-4e1c-96cf-025bb7b9d5d5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9d68adf14afe47df9c512149e4d01002)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:38.294238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0e4f69-3d32-4e1c-96cf-025bb7b9d5d5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:38.482998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-eb0e4f69-3d32-4e1c-96cf-025bb7b9d5d5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9d68adf14afe47df9c512149e4d01002', 'request_id': 'req-eb0e4f69-3d32-4e1c-96cf-025bb7b9d5d5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:38.502324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-eb0e4f69-3d32-4e1c-96cf-025bb7b9d5d5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9d68adf14afe47df9c512149e4d01002', 'request_id': 'req-eb0e4f69-3d32-4e1c-96cf-025bb7b9d5d5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:38.515653 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-eb0e4f69-3d32-4e1c-96cf-025bb7b9d5d5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9d68adf14afe47df9c512149e4d01002', 'request_id': 'req-eb0e4f69-3d32-4e1c-96cf-025bb7b9d5d5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:38.559426 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-eb0e4f69-3d32-4e1c-96cf-025bb7b9d5d5 admin admin] The token cache is being invalidate because project 9d68adf14afe47df9c512149e4d01002 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:38.560255 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-eb0e4f69-3d32-4e1c-96cf-025bb7b9d5d5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:38.561030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-eb0e4f69-3d32-4e1c-96cf-025bb7b9d5d5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:38.564597 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 889/1787] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:37 2022] DELETE /identity/v3/projects/9d68adf14afe47df9c512149e4d01002 => generated 0 bytes in 664 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:38.569390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c80da43-dbdc-4674-929c-6d27190c78e9 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:38.569832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c80da43-dbdc-4674-929c-6d27190c78e9 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:38.569937 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c80da43-dbdc-4674-929c-6d27190c78e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:38.645029 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-3c80da43-dbdc-4674-929c-6d27190c78e9 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:38.652134 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-3c80da43-dbdc-4674-929c-6d27190c78e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:38.921173 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 899/1788] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:38.930684 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cefc52b-dac8-4358-86c2-f9cc3eba55a8 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:38.931970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cefc52b-dac8-4358-86c2-f9cc3eba55a8 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:38.932413 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cefc52b-dac8-4358-86c2-f9cc3eba55a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:38.991405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3cefc52b-dac8-4358-86c2-f9cc3eba55a8 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:38.996668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3cefc52b-dac8-4358-86c2-f9cc3eba55a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:39.026268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 890/1789] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:39.030194 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c80da43-dbdc-4674-929c-6d27190c78e9 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:39.043992 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c80da43-dbdc-4674-929c-6d27190c78e9 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:39.060385 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555a5aaf-ecc0-468e-9188-c5527474ebfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:39.061257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555a5aaf-ecc0-468e-9188-c5527474ebfb admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:39.061577 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555a5aaf-ecc0-468e-9188-c5527474ebfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:39.061852 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555a5aaf-ecc0-468e-9188-c5527474ebfb admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:39.062711 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555a5aaf-ecc0-468e-9188-c5527474ebfb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:39.063313 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555a5aaf-ecc0-468e-9188-c5527474ebfb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:39.076584 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555a5aaf-ecc0-468e-9188-c5527474ebfb admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:39.100660 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cefc52b-dac8-4358-86c2-f9cc3eba55a8 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:39.102553 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3cefc52b-dac8-4358-86c2-f9cc3eba55a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:39.138091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cefc52b-dac8-4358-86c2-f9cc3eba55a8 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:39.140631 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 900/1790] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:26:39.147724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555a5aaf-ecc0-468e-9188-c5527474ebfb admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:39.154743 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61444402-b12a-40f2-b8a8-a2830809194c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:39.155459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61444402-b12a-40f2-b8a8-a2830809194c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:39.157920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61444402-b12a-40f2-b8a8-a2830809194c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:39.158341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61444402-b12a-40f2-b8a8-a2830809194c admin admin] PATH_INFO: `/v3/users/88a5946efd374183b16197f499e2dfa3` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:39.160337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555a5aaf-ecc0-468e-9188-c5527474ebfb admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:39.162039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61444402-b12a-40f2-b8a8-a2830809194c admin admin] RBAC: Authorizing `identity:delete_user(user_id=88a5946efd374183b16197f499e2dfa3)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:39.165742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95ed3cd-b664-4025-a18b-e392a0737e64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:39.166568 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95ed3cd-b664-4025-a18b-e392a0737e64 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:39.167358 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95ed3cd-b664-4025-a18b-e392a0737e64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:39.167780 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95ed3cd-b664-4025-a18b-e392a0737e64 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:39.168741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95ed3cd-b664-4025-a18b-e392a0737e64 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:39.170136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61444402-b12a-40f2-b8a8-a2830809194c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:39.171567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95ed3cd-b664-4025-a18b-e392a0737e64 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:39.214270 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:39.235679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 901/1791] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:39 2022] POST /identity/v3/projects => generated 369 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:39.241988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95ed3cd-b664-4025-a18b-e392a0737e64 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:39.318183 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-61444402-b12a-40f2-b8a8-a2830809194c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '88a5946efd374183b16197f499e2dfa3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:39.348024 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-61444402-b12a-40f2-b8a8-a2830809194c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '88a5946efd374183b16197f499e2dfa3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:39.366853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-61444402-b12a-40f2-b8a8-a2830809194c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '88a5946efd374183b16197f499e2dfa3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:39.388944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-61444402-b12a-40f2-b8a8-a2830809194c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '88a5946efd374183b16197f499e2dfa3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:39.389462 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-61444402-b12a-40f2-b8a8-a2830809194c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '88a5946efd374183b16197f499e2dfa3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:39.403299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 891/1792] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:39 2022] DELETE /identity/v3/users/88a5946efd374183b16197f499e2dfa3 => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:39.427030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95ed3cd-b664-4025-a18b-e392a0737e64 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:39.555744 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:39.591385 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a8c14c-425b-4464-b4e4-1edebd5de6fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:39.592121 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a8c14c-425b-4464-b4e4-1edebd5de6fa admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:39.592290 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a8c14c-425b-4464-b4e4-1edebd5de6fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:39.592728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a8c14c-425b-4464-b4e4-1edebd5de6fa admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:39.593609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a8c14c-425b-4464-b4e4-1edebd5de6fa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:39.599168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a8c14c-425b-4464-b4e4-1edebd5de6fa admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:39.651189 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 902/1793] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:39 2022] POST /identity/v3/users => generated 285 bytes in 410 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:39.656188 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61444402-b12a-40f2-b8a8-a2830809194c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:39.657829 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-61444402-b12a-40f2-b8a8-a2830809194c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:39.683028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61444402-b12a-40f2-b8a8-a2830809194c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:39.696812 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ae2bbd-9d35-4f16-936d-3888db23f2eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:39.699581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ae2bbd-9d35-4f16-936d-3888db23f2eb admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:39.699875 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ae2bbd-9d35-4f16-936d-3888db23f2eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:39.700154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ae2bbd-9d35-4f16-936d-3888db23f2eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:39.701757 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ae2bbd-9d35-4f16-936d-3888db23f2eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:39.706889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ae2bbd-9d35-4f16-936d-3888db23f2eb admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:39.736030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 892/1794] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:39 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:39.740656 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a8c14c-425b-4464-b4e4-1edebd5de6fa admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:39.767559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a8c14c-425b-4464-b4e4-1edebd5de6fa admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:39.783621 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566a5d46-e811-4b19-ba2b-257770781eb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:39.784557 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566a5d46-e811-4b19-ba2b-257770781eb9 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:39.784998 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566a5d46-e811-4b19-ba2b-257770781eb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:39.785465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566a5d46-e811-4b19-ba2b-257770781eb9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:39.786279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566a5d46-e811-4b19-ba2b-257770781eb9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:39.799439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566a5d46-e811-4b19-ba2b-257770781eb9 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:39.836653 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 903/1795] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:39 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:26:39.841725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ae2bbd-9d35-4f16-936d-3888db23f2eb admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:39.881661 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ae2bbd-9d35-4f16-936d-3888db23f2eb admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:39.917995 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed1a08b-ea6e-4e2c-940a-5a7edfc1aa18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:39.928089 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed1a08b-ea6e-4e2c-940a-5a7edfc1aa18 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:39.930565 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed1a08b-ea6e-4e2c-940a-5a7edfc1aa18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:39.931019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed1a08b-ea6e-4e2c-940a-5a7edfc1aa18 admin admin] PATH_INFO: `/v3/projects/f1ae139541d74da1891a729df50bcc80/users/b021b4ff906844608fea39c7e92cd503/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:39.993642 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed1a08b-ea6e-4e2c-940a-5a7edfc1aa18 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1ae139541d74da1891a729df50bcc80, user_id=b021b4ff906844608fea39c7e92cd503, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:40.015069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed1a08b-ea6e-4e2c-940a-5a7edfc1aa18 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:40.054545 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 893/1796] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:39 2022] PUT /identity/v3/projects/f1ae139541d74da1891a729df50bcc80/users/b021b4ff906844608fea39c7e92cd503/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:40.065238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d122ea7a-a9e0-4b15-a116-c37ff7bf05cb None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:40.066012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d122ea7a-a9e0-4b15-a116-c37ff7bf05cb None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:40.066546 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d122ea7a-a9e0-4b15-a116-c37ff7bf05cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:40.229166 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-d122ea7a-a9e0-4b15-a116-c37ff7bf05cb None None] MFA Rules not processed for user `b021b4ff906844608fea39c7e92cd503`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:40.232401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d122ea7a-a9e0-4b15-a116-c37ff7bf05cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:40.428218 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b619d04-0521-4710-9875-6a00ab3a4ab4 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:40.428781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b619d04-0521-4710-9875-6a00ab3a4ab4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:40.429249 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b619d04-0521-4710-9875-6a00ab3a4ab4 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:40.435623 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 894/1797] 10.222.0.85 () {54 vars in 921 bytes} [Tue Aug 30 04:26:40 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:26:40.501126 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 904/1798] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:40 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 437 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:40.504021 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b619d04-0521-4710-9875-6a00ab3a4ab4 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:40.529963 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a485994-a601-4908-94d2-d8e57aad169f None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:40.530352 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a485994-a601-4908-94d2-d8e57aad169f None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:40.530750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a485994-a601-4908-94d2-d8e57aad169f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:40.610479 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-8a485994-a601-4908-94d2-d8e57aad169f None None] MFA Rules not processed for user `4bc4d455ebf041d488b1199ca58358a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:40.615591 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8a485994-a601-4908-94d2-d8e57aad169f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:40.652501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b619d04-0521-4710-9875-6a00ab3a4ab4 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:40.667118 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba174416-6e93-4d01-b501-c8acfd43da4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:40.667746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba174416-6e93-4d01-b501-c8acfd43da4a admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:40.668167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba174416-6e93-4d01-b501-c8acfd43da4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:40.668607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba174416-6e93-4d01-b501-c8acfd43da4a admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:40.670010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba174416-6e93-4d01-b501-c8acfd43da4a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:40.672289 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba174416-6e93-4d01-b501-c8acfd43da4a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:40.743720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 895/1799] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:40 2022] POST /identity/v3/projects => generated 367 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:40.747648 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba174416-6e93-4d01-b501-c8acfd43da4a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:40.791645 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 905/1800] 10.222.0.85 () {58 vars in 1018 bytes} [Tue Aug 30 04:26:40 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:40.859392 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=88,keystone:DELETE=9,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:40.860387 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5ab14b-ebdd-40cd-9dd0-c29a08b4396a None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:40.860387 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5ab14b-ebdd-40cd-9dd0-c29a08b4396a None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:40.860703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5ab14b-ebdd-40cd-9dd0-c29a08b4396a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:40.894510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba174416-6e93-4d01-b501-c8acfd43da4a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:40.906847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b116494d-422b-4d86-a373-17cbe6bb8fc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:40.907440 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b116494d-422b-4d86-a373-17cbe6bb8fc7 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:40.907777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b116494d-422b-4d86-a373-17cbe6bb8fc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:40.908303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b116494d-422b-4d86-a373-17cbe6bb8fc7 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:40.908970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b116494d-422b-4d86-a373-17cbe6bb8fc7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:40.911104 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b116494d-422b-4d86-a373-17cbe6bb8fc7 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:40.975508 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-9b5ab14b-ebdd-40cd-9dd0-c29a08b4396a None None] MFA Rules not processed for user `f6ff88ebcb8440e99539b0a2cda8e152`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:40.977434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-9b5ab14b-ebdd-40cd-9dd0-c29a08b4396a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:40.979631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 896/1801] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:40 2022] POST /identity/v3/users => generated 290 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:40.982149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b116494d-422b-4d86-a373-17cbe6bb8fc7 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:40.985932 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b116494d-422b-4d86-a373-17cbe6bb8fc7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:41.020859 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b116494d-422b-4d86-a373-17cbe6bb8fc7 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:41.035446 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7ec36e-1316-44d1-912e-19986efe3092 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:41.036664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7ec36e-1316-44d1-912e-19986efe3092 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:41.036664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7ec36e-1316-44d1-912e-19986efe3092 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:41.037155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7ec36e-1316-44d1-912e-19986efe3092 admin admin] PATH_INFO: `/v3/projects/8ed8492885cb49dc93bedfe390643141` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:41.038488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d7ec36e-1316-44d1-912e-19986efe3092 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ed8492885cb49dc93bedfe390643141)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:41.042680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d7ec36e-1316-44d1-912e-19986efe3092 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:41.092889 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:41.178154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4d7ec36e-1316-44d1-912e-19986efe3092 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ed8492885cb49dc93bedfe390643141', 'request_id': 'req-4d7ec36e-1316-44d1-912e-19986efe3092'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:41.185987 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 906/1802] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:40 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 327 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:41.187122 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5ab14b-ebdd-40cd-9dd0-c29a08b4396a None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:41.192907 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4d7ec36e-1316-44d1-912e-19986efe3092 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ed8492885cb49dc93bedfe390643141', 'request_id': 'req-4d7ec36e-1316-44d1-912e-19986efe3092'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:41.201813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5ab14b-ebdd-40cd-9dd0-c29a08b4396a None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:41.208746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4d7ec36e-1316-44d1-912e-19986efe3092 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ed8492885cb49dc93bedfe390643141', 'request_id': 'req-4d7ec36e-1316-44d1-912e-19986efe3092'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:41.229357 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4d7ec36e-1316-44d1-912e-19986efe3092 admin admin] The token cache is being invalidate because project 8ed8492885cb49dc93bedfe390643141 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:41.229742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4d7ec36e-1316-44d1-912e-19986efe3092 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:41.230614 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4d7ec36e-1316-44d1-912e-19986efe3092 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:41.234443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 897/1803] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:40 2022] DELETE /identity/v3/projects/8ed8492885cb49dc93bedfe390643141 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:41.234443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7ec36e-1316-44d1-912e-19986efe3092 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:41.238876 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4d7ec36e-1316-44d1-912e-19986efe3092 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:41.414469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd14ed9-110c-4f28-9515-d714e4dc3a56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:41.415109 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd14ed9-110c-4f28-9515-d714e4dc3a56 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:41.415394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd14ed9-110c-4f28-9515-d714e4dc3a56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:41.415669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd14ed9-110c-4f28-9515-d714e4dc3a56 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:41.419839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd14ed9-110c-4f28-9515-d714e4dc3a56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:41.428566 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7ec36e-1316-44d1-912e-19986efe3092 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:41.435944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd14ed9-110c-4f28-9515-d714e4dc3a56 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:41.451336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 907/1804] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:41 2022] GET /identity/v3/roles => generated 1420 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:41.453359 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d16ccfa-4a04-428e-af42-e7d03df14b47 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:41.461427 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d16ccfa-4a04-428e-af42-e7d03df14b47 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:41.462182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d16ccfa-4a04-428e-af42-e7d03df14b47 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:41.463975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 908/1805] 10.222.0.85 () {54 vars in 921 bytes} [Tue Aug 30 04:26:41 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:26:41.465232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d16ccfa-4a04-428e-af42-e7d03df14b47 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:41.465861 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57893eaf-63b0-420f-9706-c7be01b5134d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:41.466503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57893eaf-63b0-420f-9706-c7be01b5134d service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:41.466787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57893eaf-63b0-420f-9706-c7be01b5134d service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:41.467061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57893eaf-63b0-420f-9706-c7be01b5134d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:41.467383 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8d16ccfa-4a04-428e-af42-e7d03df14b47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:41.471157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-57893eaf-63b0-420f-9706-c7be01b5134d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:41.486652 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d16ccfa-4a04-428e-af42-e7d03df14b47 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:41.493202 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59a79cd-d359-4c4d-a678-e69f925ad93a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:41.496488 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59a79cd-d359-4c4d-a678-e69f925ad93a admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:41.496488 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59a79cd-d359-4c4d-a678-e69f925ad93a admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:41.496488 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59a79cd-d359-4c4d-a678-e69f925ad93a admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:41.496488 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59a79cd-d359-4c4d-a678-e69f925ad93a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:41.499038 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59a79cd-d359-4c4d-a678-e69f925ad93a admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:41.508838 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57893eaf-63b0-420f-9706-c7be01b5134d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:41.510902 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57893eaf-63b0-420f-9706-c7be01b5134d service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:41.524356 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 898/1806] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:41.525406 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57893eaf-63b0-420f-9706-c7be01b5134d service nova] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:41.526840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-57893eaf-63b0-420f-9706-c7be01b5134d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:41.541468 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57893eaf-63b0-420f-9706-c7be01b5134d service nova] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:41.547961 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9dab12-cf94-426f-a505-7287a4ad78ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:41.548611 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9dab12-cf94-426f-a505-7287a4ad78ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:41.549203 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9dab12-cf94-426f-a505-7287a4ad78ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:41.550061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9dab12-cf94-426f-a505-7287a4ad78ac admin admin] PATH_INFO: `/v3/users/42ba03515f124c1094899574cacf86a7` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:41.551885 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9dab12-cf94-426f-a505-7287a4ad78ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=42ba03515f124c1094899574cacf86a7)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:41.554327 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9dab12-cf94-426f-a505-7287a4ad78ac admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:41.569172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 909/1807] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:41 2022] POST /identity/v3/projects => generated 373 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:41.570075 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59a79cd-d359-4c4d-a678-e69f925ad93a admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:41.578665 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c59a79cd-d359-4c4d-a678-e69f925ad93a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:41.638457 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=84,keystone:INSERT=4,keystone:DELETE=12 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:41.669812 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59a79cd-d359-4c4d-a678-e69f925ad93a admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:41.678134 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1952e3-0b4e-4978-8547-7f65694572d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:41.678882 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1952e3-0b4e-4978-8547-7f65694572d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:41.679180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1952e3-0b4e-4978-8547-7f65694572d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:41.679542 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1952e3-0b4e-4978-8547-7f65694572d1 admin admin] PATH_INFO: `/v3/projects/63e43417ace14fafb25b58d009bcc1a2/users/b96482390ddd4ea4ab1ce39e01a62f4c/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:41.679964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-df9dab12-cf94-426f-a505-7287a4ad78ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '42ba03515f124c1094899574cacf86a7'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:41.700087 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1952e3-0b4e-4978-8547-7f65694572d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=63e43417ace14fafb25b58d009bcc1a2, user_id=b96482390ddd4ea4ab1ce39e01a62f4c, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:41.702939 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1952e3-0b4e-4978-8547-7f65694572d1 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:41.706360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-df9dab12-cf94-426f-a505-7287a4ad78ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '42ba03515f124c1094899574cacf86a7'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:41.728524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-df9dab12-cf94-426f-a505-7287a4ad78ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '42ba03515f124c1094899574cacf86a7'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:41.731459 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 910/1808] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:41 2022] PUT /identity/v3/projects/63e43417ace14fafb25b58d009bcc1a2/users/b96482390ddd4ea4ab1ce39e01a62f4c/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:41.741987 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e27663a-6544-4f4c-bfc6-c90e75e57656 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:41.742309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e27663a-6544-4f4c-bfc6-c90e75e57656 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:41.742589 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e27663a-6544-4f4c-bfc6-c90e75e57656 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:41.744397 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-df9dab12-cf94-426f-a505-7287a4ad78ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '42ba03515f124c1094899574cacf86a7'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:41.745414 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-df9dab12-cf94-426f-a505-7287a4ad78ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '42ba03515f124c1094899574cacf86a7'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:41.752193 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 899/1809] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:41 2022] DELETE /identity/v3/users/42ba03515f124c1094899574cacf86a7 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:41.753289 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9dab12-cf94-426f-a505-7287a4ad78ac admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:41.754725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-df9dab12-cf94-426f-a505-7287a4ad78ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:41.771164 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-4e27663a-6544-4f4c-bfc6-c90e75e57656 None None] MFA Rules not processed for user `4bc4d455ebf041d488b1199ca58358a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:41.772507 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4e27663a-6544-4f4c-bfc6-c90e75e57656 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:41.816984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:41.877212 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9dab12-cf94-426f-a505-7287a4ad78ac admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:41.895202 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d025905d-5972-4e33-96eb-f73976203010 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:41.895981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d025905d-5972-4e33-96eb-f73976203010 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:41.896380 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d025905d-5972-4e33-96eb-f73976203010 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:41.896758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d025905d-5972-4e33-96eb-f73976203010 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:41.897489 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d025905d-5972-4e33-96eb-f73976203010 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:41.899569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d025905d-5972-4e33-96eb-f73976203010 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:41.905199 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 911/1810] 10.222.0.85 () {58 vars in 1018 bytes} [Tue Aug 30 04:26:41 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:41.906219 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e27663a-6544-4f4c-bfc6-c90e75e57656 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:41.907636 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4e27663a-6544-4f4c-bfc6-c90e75e57656 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:41.925897 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e27663a-6544-4f4c-bfc6-c90e75e57656 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:41.931397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4d8028-9f7e-4fd4-b8cb-25dcef281360 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:41.932168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4d8028-9f7e-4fd4-b8cb-25dcef281360 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:41.932553 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4d8028-9f7e-4fd4-b8cb-25dcef281360 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:41.932971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4d8028-9f7e-4fd4-b8cb-25dcef281360 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:41.933760 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4d8028-9f7e-4fd4-b8cb-25dcef281360 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:41.935773 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4d8028-9f7e-4fd4-b8cb-25dcef281360 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:41.939024 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 900/1811] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:41 2022] POST /identity/v3/users => generated 293 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:41.939024 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d025905d-5972-4e33-96eb-f73976203010 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:41.939867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d025905d-5972-4e33-96eb-f73976203010 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:41.954242 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 912/1812] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:41 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:41.955420 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4d8028-9f7e-4fd4-b8cb-25dcef281360 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:41.959729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d025905d-5972-4e33-96eb-f73976203010 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:41.967300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b805b5d-ec34-42e4-aec1-10fb7f7cc61a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:41.967887 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b805b5d-ec34-42e4-aec1-10fb7f7cc61a service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:41.968162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b805b5d-ec34-42e4-aec1-10fb7f7cc61a service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:41.968468 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b805b5d-ec34-42e4-aec1-10fb7f7cc61a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:41.971375 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4d8028-9f7e-4fd4-b8cb-25dcef281360 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:41.979856 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0418d9a2-167c-4b1b-99de-234d7e6b1c89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:41.980827 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0418d9a2-167c-4b1b-99de-234d7e6b1c89 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:41.981302 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0418d9a2-167c-4b1b-99de-234d7e6b1c89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:41.981757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0418d9a2-167c-4b1b-99de-234d7e6b1c89 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:41.982630 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0418d9a2-167c-4b1b-99de-234d7e6b1c89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:41.984395 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b805b5d-ec34-42e4-aec1-10fb7f7cc61a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:41.984894 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0418d9a2-167c-4b1b-99de-234d7e6b1c89 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:41.986444 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b805b5d-ec34-42e4-aec1-10fb7f7cc61a service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:42.001135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 913/1813] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:41 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:42.001135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0418d9a2-167c-4b1b-99de-234d7e6b1c89 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:42.009849 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 901/1814] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 73 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:42.010685 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b805b5d-ec34-42e4-aec1-10fb7f7cc61a service nova] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:42.029830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0418d9a2-167c-4b1b-99de-234d7e6b1c89 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:42.036415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b805b5d-ec34-42e4-aec1-10fb7f7cc61a service nova] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:42.037607 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acafb794-1f7b-45f2-8d72-407cbe60796d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:42.040537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acafb794-1f7b-45f2-8d72-407cbe60796d admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:42.040537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acafb794-1f7b-45f2-8d72-407cbe60796d admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:42.040763 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acafb794-1f7b-45f2-8d72-407cbe60796d admin admin] PATH_INFO: `/v3/projects/63e43417ace14fafb25b58d009bcc1a2/users/b96482390ddd4ea4ab1ce39e01a62f4c/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:42.041583 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea9fbaf-fc58-4089-8562-f66e3f58328a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:42.042248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea9fbaf-fc58-4089-8562-f66e3f58328a admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:42.042523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea9fbaf-fc58-4089-8562-f66e3f58328a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:42.042795 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea9fbaf-fc58-4089-8562-f66e3f58328a admin admin] PATH_INFO: `/v3/projects/83c45254f5f4416eb1bc3bbba7242e4b/users/74618524351f46a7b41b7dbd3552cff3/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:42.045650 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acafb794-1f7b-45f2-8d72-407cbe60796d admin admin] RBAC: Authorizing `identity:create_grant(project_id=63e43417ace14fafb25b58d009bcc1a2, user_id=b96482390ddd4ea4ab1ce39e01a62f4c, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:42.051090 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acafb794-1f7b-45f2-8d72-407cbe60796d admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:42.059831 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea9fbaf-fc58-4089-8562-f66e3f58328a admin admin] RBAC: Authorizing `identity:create_grant(project_id=83c45254f5f4416eb1bc3bbba7242e4b, user_id=74618524351f46a7b41b7dbd3552cff3, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:42.066278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea9fbaf-fc58-4089-8562-f66e3f58328a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:42.084693 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 914/1815] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:41 2022] PUT /identity/v3/projects/63e43417ace14fafb25b58d009bcc1a2/users/b96482390ddd4ea4ab1ce39e01a62f4c/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:42.098056 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1154635a-7426-4ed8-9f3c-23d06a20b11b None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:42.098310 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1154635a-7426-4ed8-9f3c-23d06a20b11b None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:42.098517 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1154635a-7426-4ed8-9f3c-23d06a20b11b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:42.136221 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 902/1816] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:42 2022] PUT /identity/v3/projects/83c45254f5f4416eb1bc3bbba7242e4b/users/74618524351f46a7b41b7dbd3552cff3/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:42.141656 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea9fbaf-fc58-4089-8562-f66e3f58328a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:42.147729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cea9fbaf-fc58-4089-8562-f66e3f58328a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:42.181668 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-1154635a-7426-4ed8-9f3c-23d06a20b11b None None] MFA Rules not processed for user `b96482390ddd4ea4ab1ce39e01a62f4c`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:42.187119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-1154635a-7426-4ed8-9f3c-23d06a20b11b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:42.308666 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea9fbaf-fc58-4089-8562-f66e3f58328a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:42.313866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edff441-9bca-4c65-b9d4-d74823fdd94b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:42.314637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edff441-9bca-4c65-b9d4-d74823fdd94b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:42.315035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edff441-9bca-4c65-b9d4-d74823fdd94b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:42.315986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edff441-9bca-4c65-b9d4-d74823fdd94b admin admin] PATH_INFO: `/v3/projects/0e152cdcc7d042de92d06a01e0b3ec21` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:42.317554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edff441-9bca-4c65-b9d4-d74823fdd94b admin admin] RBAC: Authorizing `identity:delete_project(project_id=0e152cdcc7d042de92d06a01e0b3ec21)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:42.319943 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edff441-9bca-4c65-b9d4-d74823fdd94b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:42.327476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:42.351170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 915/1817] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:42 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:26:42.352068 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1154635a-7426-4ed8-9f3c-23d06a20b11b None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:42.369131 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1154635a-7426-4ed8-9f3c-23d06a20b11b None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:42.390362 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1edff441-9bca-4c65-b9d4-d74823fdd94b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0e152cdcc7d042de92d06a01e0b3ec21', 'request_id': 'req-1edff441-9bca-4c65-b9d4-d74823fdd94b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:42.398193 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1edff441-9bca-4c65-b9d4-d74823fdd94b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0e152cdcc7d042de92d06a01e0b3ec21', 'request_id': 'req-1edff441-9bca-4c65-b9d4-d74823fdd94b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:42.412581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1edff441-9bca-4c65-b9d4-d74823fdd94b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0e152cdcc7d042de92d06a01e0b3ec21', 'request_id': 'req-1edff441-9bca-4c65-b9d4-d74823fdd94b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:42.442354 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1edff441-9bca-4c65-b9d4-d74823fdd94b admin admin] The token cache is being invalidate because project 0e152cdcc7d042de92d06a01e0b3ec21 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:42.445204 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1edff441-9bca-4c65-b9d4-d74823fdd94b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:42.448052 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1edff441-9bca-4c65-b9d4-d74823fdd94b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:42.452803 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 903/1818] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:42 2022] DELETE /identity/v3/projects/0e152cdcc7d042de92d06a01e0b3ec21 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:42.454023 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edff441-9bca-4c65-b9d4-d74823fdd94b admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:42.458176 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1edff441-9bca-4c65-b9d4-d74823fdd94b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:42.477700 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0974db18-61c6-456d-b47f-b79bcd6b6836 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:42.478396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0974db18-61c6-456d-b47f-b79bcd6b6836 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:42.478689 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0974db18-61c6-456d-b47f-b79bcd6b6836 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:42.479204 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0974db18-61c6-456d-b47f-b79bcd6b6836 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:42.479881 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0974db18-61c6-456d-b47f-b79bcd6b6836 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:42.482529 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0974db18-61c6-456d-b47f-b79bcd6b6836 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:42.506340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 916/1819] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:42 2022] GET /identity/v3/roles => generated 1420 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:42.506933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0974db18-61c6-456d-b47f-b79bcd6b6836 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:42.515441 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-0974db18-61c6-456d-b47f-b79bcd6b6836 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:42.535419 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:42.611140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edff441-9bca-4c65-b9d4-d74823fdd94b admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:42.615588 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6481b154-c9f2-4654-b3cc-37bc56a2c772 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:42.616208 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6481b154-c9f2-4654-b3cc-37bc56a2c772 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:42.616535 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6481b154-c9f2-4654-b3cc-37bc56a2c772 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:42.616763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6481b154-c9f2-4654-b3cc-37bc56a2c772 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:42.619047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6481b154-c9f2-4654-b3cc-37bc56a2c772 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:42.650452 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0974db18-61c6-456d-b47f-b79bcd6b6836 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:42.654724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4edf457-9b99-44d7-964e-a30c44080bae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:42.655346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4edf457-9b99-44d7-964e-a30c44080bae service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:42.655631 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4edf457-9b99-44d7-964e-a30c44080bae service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:42.655969 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4edf457-9b99-44d7-964e-a30c44080bae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:42.668555 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c4edf457-9b99-44d7-964e-a30c44080bae service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:42.722745 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6481b154-c9f2-4654-b3cc-37bc56a2c772 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:42.724686 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6481b154-c9f2-4654-b3cc-37bc56a2c772 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:42.747035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 904/1820] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:42 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:42.748533 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a56b18a-da78-4733-8514-ddd6f08693d2 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:42.749638 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a56b18a-da78-4733-8514-ddd6f08693d2 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:42.750074 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a56b18a-da78-4733-8514-ddd6f08693d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:42.765266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4edf457-9b99-44d7-964e-a30c44080bae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:42.775353 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4edf457-9b99-44d7-964e-a30c44080bae service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:42.789367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 917/1821] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:42 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 283 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:42.790463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4edf457-9b99-44d7-964e-a30c44080bae service cinder] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:42.791974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c4edf457-9b99-44d7-964e-a30c44080bae service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:42.806276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3a56b18a-da78-4733-8514-ddd6f08693d2 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:42.810725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3a56b18a-da78-4733-8514-ddd6f08693d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:42.819217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4edf457-9b99-44d7-964e-a30c44080bae service cinder] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:42.824910 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 905/1822] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:42.832950 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c65b151-623b-4667-8c11-9cf462c2060d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:42.833752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417ff628-8c23-4f1f-a315-6ce76e62c3ea None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:42.834052 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417ff628-8c23-4f1f-a315-6ce76e62c3ea None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:42.834418 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c65b151-623b-4667-8c11-9cf462c2060d admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:42.834531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417ff628-8c23-4f1f-a315-6ce76e62c3ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:42.835012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c65b151-623b-4667-8c11-9cf462c2060d admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:42.835340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c65b151-623b-4667-8c11-9cf462c2060d admin admin] PATH_INFO: `/v3/projects/83c45254f5f4416eb1bc3bbba7242e4b/users/74618524351f46a7b41b7dbd3552cff3/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:42.857669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c65b151-623b-4667-8c11-9cf462c2060d admin admin] RBAC: Authorizing `identity:create_grant(project_id=83c45254f5f4416eb1bc3bbba7242e4b, user_id=74618524351f46a7b41b7dbd3552cff3, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:42.865634 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-417ff628-8c23-4f1f-a315-6ce76e62c3ea None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:42.867055 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-417ff628-8c23-4f1f-a315-6ce76e62c3ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:42.867182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c65b151-623b-4667-8c11-9cf462c2060d admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:42.888541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 906/1823] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:42.889650 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417ff628-8c23-4f1f-a315-6ce76e62c3ea None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:42.904833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 918/1824] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:42 2022] PUT /identity/v3/projects/83c45254f5f4416eb1bc3bbba7242e4b/users/74618524351f46a7b41b7dbd3552cff3/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:42.911526 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417ff628-8c23-4f1f-a315-6ce76e62c3ea None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:43.004723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:43.080454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cdf356-f53d-4dbb-aecf-6eda7f057f95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:43.084295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cdf356-f53d-4dbb-aecf-6eda7f057f95 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:43.085177 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cdf356-f53d-4dbb-aecf-6eda7f057f95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:43.085728 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cdf356-f53d-4dbb-aecf-6eda7f057f95 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:43.086925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cdf356-f53d-4dbb-aecf-6eda7f057f95 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:43.087477 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cdf356-f53d-4dbb-aecf-6eda7f057f95 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:43.089843 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cdf356-f53d-4dbb-aecf-6eda7f057f95 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:43.122024 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 907/1825] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:26:43.126673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c65b151-623b-4667-8c11-9cf462c2060d admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:43.160211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c65b151-623b-4667-8c11-9cf462c2060d admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:43.171148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a8daa5-8752-4a97-8680-e6c20fb926c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:43.173173 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a8daa5-8752-4a97-8680-e6c20fb926c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:43.173882 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a8daa5-8752-4a97-8680-e6c20fb926c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:43.175543 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a8daa5-8752-4a97-8680-e6c20fb926c7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:43.177795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a8daa5-8752-4a97-8680-e6c20fb926c7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:43.182503 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a8daa5-8752-4a97-8680-e6c20fb926c7 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:43.270395 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 919/1826] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:43 2022] POST /identity/v3/projects => generated 367 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:43.274573 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cdf356-f53d-4dbb-aecf-6eda7f057f95 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:43.289415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cdf356-f53d-4dbb-aecf-6eda7f057f95 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:43.469611 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b3223c-8b6a-400a-95d2-d23f843fb2c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:43.470244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b3223c-8b6a-400a-95d2-d23f843fb2c0 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:43.470529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b3223c-8b6a-400a-95d2-d23f843fb2c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:43.470805 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b3223c-8b6a-400a-95d2-d23f843fb2c0 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:43.471481 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b3223c-8b6a-400a-95d2-d23f843fb2c0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:43.473667 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b3223c-8b6a-400a-95d2-d23f843fb2c0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:43.520684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a8daa5-8752-4a97-8680-e6c20fb926c7 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:43.521204 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 908/1827] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:43 2022] POST /identity/v3/users => generated 284 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:43.545881 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a8daa5-8752-4a97-8680-e6c20fb926c7 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:43.570507 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c9e4ee-77c5-4488-8e0d-44837444a460 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:43.571142 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c9e4ee-77c5-4488-8e0d-44837444a460 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:43.571421 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c9e4ee-77c5-4488-8e0d-44837444a460 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:43.571691 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c9e4ee-77c5-4488-8e0d-44837444a460 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:43.572350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c9e4ee-77c5-4488-8e0d-44837444a460 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:43.578131 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c9e4ee-77c5-4488-8e0d-44837444a460 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:43.600422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 920/1828] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:43 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:26:43.604568 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b3223c-8b6a-400a-95d2-d23f843fb2c0 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:43.626877 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b3223c-8b6a-400a-95d2-d23f843fb2c0 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:43.638155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8d0f33-34c0-48e8-84cb-663e1dfb0524 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:43.638805 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8d0f33-34c0-48e8-84cb-663e1dfb0524 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:43.639145 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8d0f33-34c0-48e8-84cb-663e1dfb0524 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:43.639453 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8d0f33-34c0-48e8-84cb-663e1dfb0524 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:43.640536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8d0f33-34c0-48e8-84cb-663e1dfb0524 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:43.643860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8d0f33-34c0-48e8-84cb-663e1dfb0524 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:43.664707 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 909/1829] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:43 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:43.665905 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c9e4ee-77c5-4488-8e0d-44837444a460 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:43.681842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c9e4ee-77c5-4488-8e0d-44837444a460 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:43.694521 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f564e3-4856-4f00-b5b5-2da80fa4df38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:43.696078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f564e3-4856-4f00-b5b5-2da80fa4df38 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:43.696652 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f564e3-4856-4f00-b5b5-2da80fa4df38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:43.697142 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f564e3-4856-4f00-b5b5-2da80fa4df38 admin admin] PATH_INFO: `/v3/projects/4d13d71a5ebf4d0f966b20637fb085b8/users/584031e90f424330b7c91ac7c6839859/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:43.768530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f564e3-4856-4f00-b5b5-2da80fa4df38 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d13d71a5ebf4d0f966b20637fb085b8, user_id=584031e90f424330b7c91ac7c6839859, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:43.773396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f564e3-4856-4f00-b5b5-2da80fa4df38 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:43.806021 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 921/1830] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:43 2022] PUT /identity/v3/projects/4d13d71a5ebf4d0f966b20637fb085b8/users/584031e90f424330b7c91ac7c6839859/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:43.823940 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82fe992-2807-43e1-ae28-2150a35246ae None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:43.826106 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82fe992-2807-43e1-ae28-2150a35246ae None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:43.826478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82fe992-2807-43e1-ae28-2150a35246ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:43.890965 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-f82fe992-2807-43e1-ae28-2150a35246ae None None] MFA Rules not processed for user `584031e90f424330b7c91ac7c6839859`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:43.892399 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f82fe992-2807-43e1-ae28-2150a35246ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:44.050577 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 910/1831] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:43 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:44.057156 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f564e3-4856-4f00-b5b5-2da80fa4df38 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:44.164033 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:44.179019 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f564e3-4856-4f00-b5b5-2da80fa4df38 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:44.184997 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5f77fa-750b-407d-a6eb-1364710b8ce4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:44.186647 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5f77fa-750b-407d-a6eb-1364710b8ce4 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:44.187146 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5f77fa-750b-407d-a6eb-1364710b8ce4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:44.187540 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5f77fa-750b-407d-a6eb-1364710b8ce4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:44.189391 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5f77fa-750b-407d-a6eb-1364710b8ce4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:44.192207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5f77fa-750b-407d-a6eb-1364710b8ce4 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:44.238768 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 922/1832] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:44 2022] POST /identity/v3/projects => generated 367 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:44.242963 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82fe992-2807-43e1-ae28-2150a35246ae None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:44.262746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82fe992-2807-43e1-ae28-2150a35246ae None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:44.302123 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:44.336918 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93914d78-3cb1-456e-8b7c-db90499686e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:44.338012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93914d78-3cb1-456e-8b7c-db90499686e3 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:44.338241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93914d78-3cb1-456e-8b7c-db90499686e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:44.338611 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93914d78-3cb1-456e-8b7c-db90499686e3 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:44.339318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93914d78-3cb1-456e-8b7c-db90499686e3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:44.341469 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93914d78-3cb1-456e-8b7c-db90499686e3 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:44.388590 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 911/1833] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:44 2022] POST /identity/v3/users => generated 290 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:44.389398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5f77fa-750b-407d-a6eb-1364710b8ce4 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:44.397730 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b17e4b0-f55c-4f2a-9ae4-7835852655ed None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:44.398595 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b17e4b0-f55c-4f2a-9ae4-7835852655ed None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:44.399035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b17e4b0-f55c-4f2a-9ae4-7835852655ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:44.411696 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5f77fa-750b-407d-a6eb-1364710b8ce4 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:44.419185 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbca50a-fc68-4f5d-9d78-eec762f4c6b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:44.420158 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbca50a-fc68-4f5d-9d78-eec762f4c6b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:44.420926 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbca50a-fc68-4f5d-9d78-eec762f4c6b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:44.421466 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbca50a-fc68-4f5d-9d78-eec762f4c6b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:44.431384 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbca50a-fc68-4f5d-9d78-eec762f4c6b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:44.433745 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-1b17e4b0-f55c-4f2a-9ae4-7835852655ed None None] MFA Rules not processed for user `74618524351f46a7b41b7dbd3552cff3`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:44.435205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1b17e4b0-f55c-4f2a-9ae4-7835852655ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:44.438357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbca50a-fc68-4f5d-9d78-eec762f4c6b5 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:44.495622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 923/1834] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:44 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:26:44.498528 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbca50a-fc68-4f5d-9d78-eec762f4c6b5 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:44.591575 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbca50a-fc68-4f5d-9d78-eec762f4c6b5 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:44.613116 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 912/1835] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:44 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:44.616142 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f3f78d-5ddf-4274-bafa-9fa8cf0614a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:44.617294 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f3f78d-5ddf-4274-bafa-9fa8cf0614a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:44.617726 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f3f78d-5ddf-4274-bafa-9fa8cf0614a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:44.618188 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f3f78d-5ddf-4274-bafa-9fa8cf0614a1 admin admin] PATH_INFO: `/v3/projects/6062aa736043470b8ac278988e2dee97/users/b4e2103dd9014e76a76f268a278edad0/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:44.643725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f3f78d-5ddf-4274-bafa-9fa8cf0614a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6062aa736043470b8ac278988e2dee97, user_id=b4e2103dd9014e76a76f268a278edad0, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:44.646346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f3f78d-5ddf-4274-bafa-9fa8cf0614a1 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:44.654255 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b17e4b0-f55c-4f2a-9ae4-7835852655ed None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:44.667253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 924/1836] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:44 2022] PUT /identity/v3/projects/6062aa736043470b8ac278988e2dee97/users/b4e2103dd9014e76a76f268a278edad0/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:44.670864 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f3f78d-5ddf-4274-bafa-9fa8cf0614a1 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:44.768076 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b17e4b0-f55c-4f2a-9ae4-7835852655ed None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:44.780189 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f3f78d-5ddf-4274-bafa-9fa8cf0614a1 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:44.786297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9590388-e609-4863-ac04-39f6761b5019 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:44.786913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9590388-e609-4863-ac04-39f6761b5019 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:44.787188 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9590388-e609-4863-ac04-39f6761b5019 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:44.787459 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9590388-e609-4863-ac04-39f6761b5019 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:44.788152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9590388-e609-4863-ac04-39f6761b5019 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:44.789905 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9590388-e609-4863-ac04-39f6761b5019 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:44.811891 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 925/1837] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:44 2022] GET /identity/v3/roles => generated 1420 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:26:44.812930 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9590388-e609-4863-ac04-39f6761b5019 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:44.825013 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9590388-e609-4863-ac04-39f6761b5019 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:44.827651 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81389ec3-5b07-4a79-ae48-b9063ed2e28d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:44.828407 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81389ec3-5b07-4a79-ae48-b9063ed2e28d service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:44.828892 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81389ec3-5b07-4a79-ae48-b9063ed2e28d service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:44.829319 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81389ec3-5b07-4a79-ae48-b9063ed2e28d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:44.831285 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-81389ec3-5b07-4a79-ae48-b9063ed2e28d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:44.836097 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b70b965-21aa-407f-abf1-0ddb0dc9fb70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:44.836791 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b70b965-21aa-407f-abf1-0ddb0dc9fb70 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:44.837264 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b70b965-21aa-407f-abf1-0ddb0dc9fb70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:44.837539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b70b965-21aa-407f-abf1-0ddb0dc9fb70 admin admin] PATH_INFO: `/v3/projects/6062aa736043470b8ac278988e2dee97/users/b4e2103dd9014e76a76f268a278edad0/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:44.839690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b70b965-21aa-407f-abf1-0ddb0dc9fb70 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6062aa736043470b8ac278988e2dee97, user_id=b4e2103dd9014e76a76f268a278edad0, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:44.842941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b70b965-21aa-407f-abf1-0ddb0dc9fb70 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:44.853398 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:44.861072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 926/1838] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:44 2022] PUT /identity/v3/projects/6062aa736043470b8ac278988e2dee97/users/b4e2103dd9014e76a76f268a278edad0/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:44.871694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb80f62-26ba-4d65-8a9f-883009ccf422 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:44.873896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb80f62-26ba-4d65-8a9f-883009ccf422 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:44.874460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb80f62-26ba-4d65-8a9f-883009ccf422 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:44.907139 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-9fb80f62-26ba-4d65-8a9f-883009ccf422 None None] MFA Rules not processed for user `b4e2103dd9014e76a76f268a278edad0`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:44.908443 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-9fb80f62-26ba-4d65-8a9f-883009ccf422 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:44.913136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81389ec3-5b07-4a79-ae48-b9063ed2e28d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:44.914597 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81389ec3-5b07-4a79-ae48-b9063ed2e28d service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:44.925340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 913/1839] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:44 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 272 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:26:45.047042 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 927/1840] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:44 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:26:45.135343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81389ec3-5b07-4a79-ae48-b9063ed2e28d service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:45.146830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81389ec3-5b07-4a79-ae48-b9063ed2e28d service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:45.212231 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cb6fde-96ee-4711-b8e2-48115de26c99 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:45.213267 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cb6fde-96ee-4711-b8e2-48115de26c99 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:45.213987 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cb6fde-96ee-4711-b8e2-48115de26c99 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:45.214360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cb6fde-96ee-4711-b8e2-48115de26c99 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:45.230443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69cb6fde-96ee-4711-b8e2-48115de26c99 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:45.232405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69cb6fde-96ee-4711-b8e2-48115de26c99 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:45.243088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 914/1841] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:45 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:45.306108 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb80f62-26ba-4d65-8a9f-883009ccf422 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:45.315862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb80f62-26ba-4d65-8a9f-883009ccf422 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:45.319599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b82ef99-5610-491d-8e46-37acf5b76c2f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:45.320686 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b82ef99-5610-491d-8e46-37acf5b76c2f service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:45.320686 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b82ef99-5610-491d-8e46-37acf5b76c2f service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:45.320861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b82ef99-5610-491d-8e46-37acf5b76c2f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:45.329965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:45.394745 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b82ef99-5610-491d-8e46-37acf5b76c2f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:45.396234 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b82ef99-5610-491d-8e46-37acf5b76c2f service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:45.412131 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 928/1842] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:45 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:46.077095 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cb6fde-96ee-4711-b8e2-48115de26c99 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:46.078583 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-69cb6fde-96ee-4711-b8e2-48115de26c99 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:46.153990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cb6fde-96ee-4711-b8e2-48115de26c99 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:46.158477 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f371be99-df82-4502-986c-9b1ec18de287 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:46.159790 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f371be99-df82-4502-986c-9b1ec18de287 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:46.160064 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f371be99-df82-4502-986c-9b1ec18de287 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:46.160333 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f371be99-df82-4502-986c-9b1ec18de287 admin admin] PATH_INFO: `/v3/users/b021b4ff906844608fea39c7e92cd503` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:46.161489 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f371be99-df82-4502-986c-9b1ec18de287 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b021b4ff906844608fea39c7e92cd503)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:46.163424 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f371be99-df82-4502-986c-9b1ec18de287 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:46.210123 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f371be99-df82-4502-986c-9b1ec18de287 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b021b4ff906844608fea39c7e92cd503'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:46.220949 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:46.222322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f371be99-df82-4502-986c-9b1ec18de287 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b021b4ff906844608fea39c7e92cd503'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:46.230108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f371be99-df82-4502-986c-9b1ec18de287 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b021b4ff906844608fea39c7e92cd503'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:46.238181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f371be99-df82-4502-986c-9b1ec18de287 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b021b4ff906844608fea39c7e92cd503'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:46.239341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f371be99-df82-4502-986c-9b1ec18de287 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b021b4ff906844608fea39c7e92cd503'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:46.246069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 915/1843] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:46 2022] DELETE /identity/v3/users/b021b4ff906844608fea39c7e92cd503 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:46.250525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e05dd7a-6778-42dd-8fb5-bf997a590f52 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:46.250949 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e05dd7a-6778-42dd-8fb5-bf997a590f52 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:46.251135 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e05dd7a-6778-42dd-8fb5-bf997a590f52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:46.272325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-2e05dd7a-6778-42dd-8fb5-bf997a590f52 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:46.273555 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-2e05dd7a-6778-42dd-8fb5-bf997a590f52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:46.333701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 929/1844] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:46.343364 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f371be99-df82-4502-986c-9b1ec18de287 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:46.344772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f371be99-df82-4502-986c-9b1ec18de287 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:46.405957 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f371be99-df82-4502-986c-9b1ec18de287 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:46.412683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56942ab-3969-4ce0-8113-9157d2d30d59 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:46.413574 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56942ab-3969-4ce0-8113-9157d2d30d59 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:46.413782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56942ab-3969-4ce0-8113-9157d2d30d59 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:46.414078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56942ab-3969-4ce0-8113-9157d2d30d59 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:46.425070 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56942ab-3969-4ce0-8113-9157d2d30d59 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:46.426458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56942ab-3969-4ce0-8113-9157d2d30d59 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:46.435895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 916/1845] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:46.699052 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e05dd7a-6778-42dd-8fb5-bf997a590f52 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:46.700498 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-2e05dd7a-6778-42dd-8fb5-bf997a590f52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:46.713932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e05dd7a-6778-42dd-8fb5-bf997a590f52 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:46.720407 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5c84e3-2f58-4c4b-986e-cbad3ae75aa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:46.721068 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5c84e3-2f58-4c4b-986e-cbad3ae75aa7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:46.721354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5c84e3-2f58-4c4b-986e-cbad3ae75aa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:46.721628 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5c84e3-2f58-4c4b-986e-cbad3ae75aa7 admin admin] PATH_INFO: `/v3/projects/f1ae139541d74da1891a729df50bcc80` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:46.722757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5c84e3-2f58-4c4b-986e-cbad3ae75aa7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1ae139541d74da1891a729df50bcc80)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:46.724727 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5c84e3-2f58-4c4b-986e-cbad3ae75aa7 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:46.774204 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-5b5c84e3-2f58-4c4b-986e-cbad3ae75aa7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1ae139541d74da1891a729df50bcc80', 'request_id': 'req-5b5c84e3-2f58-4c4b-986e-cbad3ae75aa7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:46.780106 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-5b5c84e3-2f58-4c4b-986e-cbad3ae75aa7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1ae139541d74da1891a729df50bcc80', 'request_id': 'req-5b5c84e3-2f58-4c4b-986e-cbad3ae75aa7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:46.788116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-5b5c84e3-2f58-4c4b-986e-cbad3ae75aa7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1ae139541d74da1891a729df50bcc80', 'request_id': 'req-5b5c84e3-2f58-4c4b-986e-cbad3ae75aa7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:46.800410 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-5b5c84e3-2f58-4c4b-986e-cbad3ae75aa7 admin admin] The token cache is being invalidate because project f1ae139541d74da1891a729df50bcc80 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:46.800695 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-5b5c84e3-2f58-4c4b-986e-cbad3ae75aa7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:46.801494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-5b5c84e3-2f58-4c4b-986e-cbad3ae75aa7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:46.802886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 930/1846] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:46 2022] DELETE /identity/v3/projects/f1ae139541d74da1891a729df50bcc80 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:46.807031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56942ab-3969-4ce0-8113-9157d2d30d59 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:46.818415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56942ab-3969-4ce0-8113-9157d2d30d59 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:46.880794 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e20a14-7de3-4785-911c-032c0618bc00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:46.881930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e20a14-7de3-4785-911c-032c0618bc00 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:46.882198 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e20a14-7de3-4785-911c-032c0618bc00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:46.882487 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e20a14-7de3-4785-911c-032c0618bc00 admin admin] PATH_INFO: `/v3/users/b96482390ddd4ea4ab1ce39e01a62f4c` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:46.883812 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e20a14-7de3-4785-911c-032c0618bc00 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b96482390ddd4ea4ab1ce39e01a62f4c)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:46.886385 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e20a14-7de3-4785-911c-032c0618bc00 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:46.949002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-59e20a14-7de3-4785-911c-032c0618bc00 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b96482390ddd4ea4ab1ce39e01a62f4c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:46.964530 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-59e20a14-7de3-4785-911c-032c0618bc00 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b96482390ddd4ea4ab1ce39e01a62f4c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:46.972089 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-59e20a14-7de3-4785-911c-032c0618bc00 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b96482390ddd4ea4ab1ce39e01a62f4c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:46.985053 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-59e20a14-7de3-4785-911c-032c0618bc00 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b96482390ddd4ea4ab1ce39e01a62f4c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:46.989373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-59e20a14-7de3-4785-911c-032c0618bc00 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b96482390ddd4ea4ab1ce39e01a62f4c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:46.993533 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 917/1847] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:46 2022] DELETE /identity/v3/users/b96482390ddd4ea4ab1ce39e01a62f4c => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:47.339286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5c84e3-2f58-4c4b-986e-cbad3ae75aa7 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:47.340758 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-5b5c84e3-2f58-4c4b-986e-cbad3ae75aa7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:47.359983 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5c84e3-2f58-4c4b-986e-cbad3ae75aa7 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:47.392524 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:47.420436 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3accf68-1213-4395-8977-23239bb11c67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:47.422382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3accf68-1213-4395-8977-23239bb11c67 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:47.422382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3accf68-1213-4395-8977-23239bb11c67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:47.422382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3accf68-1213-4395-8977-23239bb11c67 admin admin] PATH_INFO: `/v3/projects/63e43417ace14fafb25b58d009bcc1a2` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:47.422969 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3accf68-1213-4395-8977-23239bb11c67 admin admin] RBAC: Authorizing `identity:delete_project(project_id=63e43417ace14fafb25b58d009bcc1a2)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:47.425504 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3accf68-1213-4395-8977-23239bb11c67 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:47.518535 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f3accf68-1213-4395-8977-23239bb11c67 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '63e43417ace14fafb25b58d009bcc1a2', 'request_id': 'req-f3accf68-1213-4395-8977-23239bb11c67'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:47.525310 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f3accf68-1213-4395-8977-23239bb11c67 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '63e43417ace14fafb25b58d009bcc1a2', 'request_id': 'req-f3accf68-1213-4395-8977-23239bb11c67'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:47.535436 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f3accf68-1213-4395-8977-23239bb11c67 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '63e43417ace14fafb25b58d009bcc1a2', 'request_id': 'req-f3accf68-1213-4395-8977-23239bb11c67'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:47.550311 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f3accf68-1213-4395-8977-23239bb11c67 admin admin] The token cache is being invalidate because project 63e43417ace14fafb25b58d009bcc1a2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:47.550607 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f3accf68-1213-4395-8977-23239bb11c67 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:47.551278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f3accf68-1213-4395-8977-23239bb11c67 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:47.553310 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 931/1848] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:47 2022] DELETE /identity/v3/projects/63e43417ace14fafb25b58d009bcc1a2 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:47.557976 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607caad-492d-4d46-8479-3eea4e34f7ef None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:47.558841 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607caad-492d-4d46-8479-3eea4e34f7ef None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:47.559481 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607caad-492d-4d46-8479-3eea4e34f7ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:47.572542 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:47.593074 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-e607caad-492d-4d46-8479-3eea4e34f7ef None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:47.594537 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e607caad-492d-4d46-8479-3eea4e34f7ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:47.735834 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 918/1849] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:47.740247 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41341259-4d36-4d8b-ba3e-8c9b91e50dcc None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:47.740745 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41341259-4d36-4d8b-ba3e-8c9b91e50dcc None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:47.741640 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41341259-4d36-4d8b-ba3e-8c9b91e50dcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:47.767204 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-41341259-4d36-4d8b-ba3e-8c9b91e50dcc None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:47.768440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-41341259-4d36-4d8b-ba3e-8c9b91e50dcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:47.779380 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 932/1850] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:47.790321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e607caad-492d-4d46-8479-3eea4e34f7ef None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:47.803440 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e607caad-492d-4d46-8479-3eea4e34f7ef None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:47.810444 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e75c973-96c3-4434-9c99-47f2f824bdc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:47.811270 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e75c973-96c3-4434-9c99-47f2f824bdc7 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:47.811721 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e75c973-96c3-4434-9c99-47f2f824bdc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:47.812009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e75c973-96c3-4434-9c99-47f2f824bdc7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:47.812673 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e75c973-96c3-4434-9c99-47f2f824bdc7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:47.812982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e75c973-96c3-4434-9c99-47f2f824bdc7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:47.814635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e75c973-96c3-4434-9c99-47f2f824bdc7 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:47.827147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 919/1851] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:26:47.831751 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41341259-4d36-4d8b-ba3e-8c9b91e50dcc None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:47.841446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41341259-4d36-4d8b-ba3e-8c9b91e50dcc None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:47.846169 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b766ab5-3b0f-4cfc-afd7-6dc98963099f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:47.847294 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b766ab5-3b0f-4cfc-afd7-6dc98963099f admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:47.847568 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b766ab5-3b0f-4cfc-afd7-6dc98963099f admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:47.847830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b766ab5-3b0f-4cfc-afd7-6dc98963099f admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:47.848462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b766ab5-3b0f-4cfc-afd7-6dc98963099f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:47.850370 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b766ab5-3b0f-4cfc-afd7-6dc98963099f admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:47.884252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 933/1852] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:47 2022] POST /identity/v3/projects => generated 387 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:26:47.885497 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e75c973-96c3-4434-9c99-47f2f824bdc7 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:47.897277 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e75c973-96c3-4434-9c99-47f2f824bdc7 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:47.957347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b739bb-64c7-40ac-a7dd-f879cc039095 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:47.957895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b739bb-64c7-40ac-a7dd-f879cc039095 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:47.958169 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b739bb-64c7-40ac-a7dd-f879cc039095 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:47.958436 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b739bb-64c7-40ac-a7dd-f879cc039095 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:47.959274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b739bb-64c7-40ac-a7dd-f879cc039095 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:47.961506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b739bb-64c7-40ac-a7dd-f879cc039095 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:47.997902 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 920/1853] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:47 2022] POST /identity/v3/users => generated 294 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:26:48.000221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b766ab5-3b0f-4cfc-afd7-6dc98963099f admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:48.019005 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b766ab5-3b0f-4cfc-afd7-6dc98963099f admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:48.025324 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97106c7-c678-4808-baee-5195522ddbbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:48.026038 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97106c7-c678-4808-baee-5195522ddbbe admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:48.026303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97106c7-c678-4808-baee-5195522ddbbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:48.026578 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97106c7-c678-4808-baee-5195522ddbbe admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:48.027231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97106c7-c678-4808-baee-5195522ddbbe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:48.028931 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97106c7-c678-4808-baee-5195522ddbbe admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:48.040250 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 934/1854] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:47 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:26:48.044357 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b739bb-64c7-40ac-a7dd-f879cc039095 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:48.055884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b739bb-64c7-40ac-a7dd-f879cc039095 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:48.061196 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cedd1ef-ceb5-4155-ba2a-6b52c0da6558 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:48.061884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cedd1ef-ceb5-4155-ba2a-6b52c0da6558 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:48.062163 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cedd1ef-ceb5-4155-ba2a-6b52c0da6558 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:48.062429 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cedd1ef-ceb5-4155-ba2a-6b52c0da6558 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:48.063071 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cedd1ef-ceb5-4155-ba2a-6b52c0da6558 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:48.064893 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cedd1ef-ceb5-4155-ba2a-6b52c0da6558 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:48.113842 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 921/1855] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:48 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:48.119281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97106c7-c678-4808-baee-5195522ddbbe admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:48.151284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97106c7-c678-4808-baee-5195522ddbbe admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:48.166875 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ec8a8c-7573-4ef1-a3d0-5648fd6aecc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:48.166875 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ec8a8c-7573-4ef1-a3d0-5648fd6aecc0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:48.167195 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ec8a8c-7573-4ef1-a3d0-5648fd6aecc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:48.167454 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ec8a8c-7573-4ef1-a3d0-5648fd6aecc0 admin admin] PATH_INFO: `/v3/projects/6ec4e0fecacd44fd9e7a5b5db410ddbb/users/f0161474bff54c57b95c2524d75197b5/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:48.201860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ec8a8c-7573-4ef1-a3d0-5648fd6aecc0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ec4e0fecacd44fd9e7a5b5db410ddbb, user_id=f0161474bff54c57b95c2524d75197b5, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:48.204883 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ec8a8c-7573-4ef1-a3d0-5648fd6aecc0 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:48.233681 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 935/1856] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:48 2022] PUT /identity/v3/projects/6ec4e0fecacd44fd9e7a5b5db410ddbb/users/f0161474bff54c57b95c2524d75197b5/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:48.245153 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b3f2b4-4779-45b1-a1f3-8f1066914053 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:48.246021 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b3f2b4-4779-45b1-a1f3-8f1066914053 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:48.246497 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b3f2b4-4779-45b1-a1f3-8f1066914053 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:48.314439 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-78b3f2b4-4779-45b1-a1f3-8f1066914053 None None] MFA Rules not processed for user `f0161474bff54c57b95c2524d75197b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:48.318669 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-78b3f2b4-4779-45b1-a1f3-8f1066914053 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:48.482117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:48.498290 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 922/1857] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:48 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:48.503589 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ec8a8c-7573-4ef1-a3d0-5648fd6aecc0 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:48.650112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b3f2b4-4779-45b1-a1f3-8f1066914053 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:48.653049 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-78b3f2b4-4779-45b1-a1f3-8f1066914053 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:48.663692 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ec8a8c-7573-4ef1-a3d0-5648fd6aecc0 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:48.682710 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546e5a76-251f-42a9-a967-6391846a18a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:48.683514 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546e5a76-251f-42a9-a967-6391846a18a4 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:48.683836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546e5a76-251f-42a9-a967-6391846a18a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:48.684233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546e5a76-251f-42a9-a967-6391846a18a4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:48.685205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546e5a76-251f-42a9-a967-6391846a18a4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:48.687353 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546e5a76-251f-42a9-a967-6391846a18a4 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:48.697749 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:48.740211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 936/1858] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:48 2022] POST /identity/v3/projects => generated 385 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:48.744343 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546e5a76-251f-42a9-a967-6391846a18a4 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:48.812796 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b3f2b4-4779-45b1-a1f3-8f1066914053 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:48.846499 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546e5a76-251f-42a9-a967-6391846a18a4 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:48.866423 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9fd2cd-9c5c-4d60-bf19-efec5e88b2eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:48.868864 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9fd2cd-9c5c-4d60-bf19-efec5e88b2eb admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:48.871481 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9fd2cd-9c5c-4d60-bf19-efec5e88b2eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:48.871868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9fd2cd-9c5c-4d60-bf19-efec5e88b2eb admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:48.872677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9fd2cd-9c5c-4d60-bf19-efec5e88b2eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:48.878394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9fd2cd-9c5c-4d60-bf19-efec5e88b2eb admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:48.929009 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 937/1859] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:48 2022] POST /identity/v3/users => generated 299 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:48.933327 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9fd2cd-9c5c-4d60-bf19-efec5e88b2eb admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:48.935481 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d744a44-5899-478c-91b9-9662dd1d88fc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:48.936473 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d744a44-5899-478c-91b9-9662dd1d88fc service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:48.936838 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d744a44-5899-478c-91b9-9662dd1d88fc service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:48.937261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d744a44-5899-478c-91b9-9662dd1d88fc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:48.939864 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0d744a44-5899-478c-91b9-9662dd1d88fc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:48.960095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9fd2cd-9c5c-4d60-bf19-efec5e88b2eb admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:48.984817 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517c5179-af2b-40cd-9338-a1e5ad8ca37f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:48.985229 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517c5179-af2b-40cd-9338-a1e5ad8ca37f admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:48.986162 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517c5179-af2b-40cd-9338-a1e5ad8ca37f admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:48.990013 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517c5179-af2b-40cd-9338-a1e5ad8ca37f admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:48.995400 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517c5179-af2b-40cd-9338-a1e5ad8ca37f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:49.001317 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517c5179-af2b-40cd-9338-a1e5ad8ca37f admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:49.057074 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 938/1860] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:48 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:26:49.067805 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517c5179-af2b-40cd-9338-a1e5ad8ca37f admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:49.096011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517c5179-af2b-40cd-9338-a1e5ad8ca37f admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:49.110709 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11eaf33c-615b-4157-aeb8-a000c73660ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:49.112257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11eaf33c-615b-4157-aeb8-a000c73660ac admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:49.112558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11eaf33c-615b-4157-aeb8-a000c73660ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:49.112984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11eaf33c-615b-4157-aeb8-a000c73660ac admin admin] PATH_INFO: `/v3/projects/dc2d44e62d6d4e508863ce69a39ca0c4/users/b95cf705ee0841a5b07ad7b5424cf722/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:49.136286 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d744a44-5899-478c-91b9-9662dd1d88fc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:49.139141 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d744a44-5899-478c-91b9-9662dd1d88fc service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:49.151853 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11eaf33c-615b-4157-aeb8-a000c73660ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc2d44e62d6d4e508863ce69a39ca0c4, user_id=b95cf705ee0841a5b07ad7b5424cf722, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:49.155815 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 923/1861] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:48 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 507 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:49.172896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11eaf33c-615b-4157-aeb8-a000c73660ac admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:49.207360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 939/1862] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:49 2022] PUT /identity/v3/projects/dc2d44e62d6d4e508863ce69a39ca0c4/users/b95cf705ee0841a5b07ad7b5424cf722/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:49.208324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d744a44-5899-478c-91b9-9662dd1d88fc service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:49.239903 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d744a44-5899-478c-91b9-9662dd1d88fc service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:49.354199 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d016c091-5d6f-4acb-ae9a-a4a268ab7bf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:49.355694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d016c091-5d6f-4acb-ae9a-a4a268ab7bf1 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:49.357303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d016c091-5d6f-4acb-ae9a-a4a268ab7bf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:49.358197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d016c091-5d6f-4acb-ae9a-a4a268ab7bf1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:49.359069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d016c091-5d6f-4acb-ae9a-a4a268ab7bf1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:49.361019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d016c091-5d6f-4acb-ae9a-a4a268ab7bf1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:49.376718 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:49.391131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 924/1863] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:49 2022] GET /identity/v3/roles => generated 1420 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:49.396610 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11eaf33c-615b-4157-aeb8-a000c73660ac admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:49.411811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11eaf33c-615b-4157-aeb8-a000c73660ac admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:49.424080 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb01866-c4f1-4c0e-908e-8eeec25fe8e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:49.425240 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb01866-c4f1-4c0e-908e-8eeec25fe8e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:49.425240 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb01866-c4f1-4c0e-908e-8eeec25fe8e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:49.425527 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb01866-c4f1-4c0e-908e-8eeec25fe8e6 admin admin] PATH_INFO: `/v3/projects/dc2d44e62d6d4e508863ce69a39ca0c4/users/b95cf705ee0841a5b07ad7b5424cf722/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:49.429412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb01866-c4f1-4c0e-908e-8eeec25fe8e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc2d44e62d6d4e508863ce69a39ca0c4, user_id=b95cf705ee0841a5b07ad7b5424cf722, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:49.433603 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb01866-c4f1-4c0e-908e-8eeec25fe8e6 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:49.463687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 940/1864] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:49 2022] PUT /identity/v3/projects/dc2d44e62d6d4e508863ce69a39ca0c4/users/b95cf705ee0841a5b07ad7b5424cf722/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:49.476150 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bf48f1-e72e-4b7e-ab42-e2a6eb2eb338 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:49.476646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bf48f1-e72e-4b7e-ab42-e2a6eb2eb338 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:49.476769 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bf48f1-e72e-4b7e-ab42-e2a6eb2eb338 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:49.515673 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-18bf48f1-e72e-4b7e-ab42-e2a6eb2eb338 None None] MFA Rules not processed for user `b95cf705ee0841a5b07ad7b5424cf722`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:49.517351 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-18bf48f1-e72e-4b7e-ab42-e2a6eb2eb338 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:49.606114 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 925/1865] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:49 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:49.687960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb01866-c4f1-4c0e-908e-8eeec25fe8e6 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:49.728221 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bf48f1-e72e-4b7e-ab42-e2a6eb2eb338 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:49.730784 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-18bf48f1-e72e-4b7e-ab42-e2a6eb2eb338 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:49.805212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb01866-c4f1-4c0e-908e-8eeec25fe8e6 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:49.809157 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1391b94-4dc9-47d1-b1a3-e6f333e9d0ef service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:49.809932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1391b94-4dc9-47d1-b1a3-e6f333e9d0ef service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:49.810213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1391b94-4dc9-47d1-b1a3-e6f333e9d0ef service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:49.810690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1391b94-4dc9-47d1-b1a3-e6f333e9d0ef service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:49.823289 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1391b94-4dc9-47d1-b1a3-e6f333e9d0ef service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:49.824933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1391b94-4dc9-47d1-b1a3-e6f333e9d0ef service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:49.838639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 941/1866] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:49 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:49.873344 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1391b94-4dc9-47d1-b1a3-e6f333e9d0ef service cinder] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:49.892162 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1391b94-4dc9-47d1-b1a3-e6f333e9d0ef service cinder] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:49.895493 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bf48f1-e72e-4b7e-ab42-e2a6eb2eb338 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:49.901194 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376c8315-7a93-4616-b6b5-06ac6bd71a3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:49.903840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376c8315-7a93-4616-b6b5-06ac6bd71a3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:49.904580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376c8315-7a93-4616-b6b5-06ac6bd71a3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:49.904856 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376c8315-7a93-4616-b6b5-06ac6bd71a3a admin admin] PATH_INFO: `/v3/users/f0161474bff54c57b95c2524d75197b5` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:49.905728 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5d7a25-0f50-4865-baca-1792e2e83fa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:49.906101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376c8315-7a93-4616-b6b5-06ac6bd71a3a admin admin] RBAC: Authorizing `identity:delete_user(user_id=f0161474bff54c57b95c2524d75197b5)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:49.906891 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5d7a25-0f50-4865-baca-1792e2e83fa1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:49.908619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5d7a25-0f50-4865-baca-1792e2e83fa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:49.909013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5d7a25-0f50-4865-baca-1792e2e83fa1 admin admin] PATH_INFO: `/v3/users/584031e90f424330b7c91ac7c6839859` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:49.920267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376c8315-7a93-4616-b6b5-06ac6bd71a3a admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:49.920741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5d7a25-0f50-4865-baca-1792e2e83fa1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=584031e90f424330b7c91ac7c6839859)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:49.920741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5d7a25-0f50-4865-baca-1792e2e83fa1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:49.959928 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=90,keystone:INSERT=6,keystone:DELETE=4 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:50.018918 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-376c8315-7a93-4616-b6b5-06ac6bd71a3a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f0161474bff54c57b95c2524d75197b5'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:50.019592 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ce5d7a25-0f50-4865-baca-1792e2e83fa1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '584031e90f424330b7c91ac7c6839859'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:50.030894 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-376c8315-7a93-4616-b6b5-06ac6bd71a3a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f0161474bff54c57b95c2524d75197b5'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:50.032200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-376c8315-7a93-4616-b6b5-06ac6bd71a3a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f0161474bff54c57b95c2524d75197b5'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:50.061253 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ce5d7a25-0f50-4865-baca-1792e2e83fa1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '584031e90f424330b7c91ac7c6839859'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:50.061703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-376c8315-7a93-4616-b6b5-06ac6bd71a3a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f0161474bff54c57b95c2524d75197b5'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:50.064700 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-376c8315-7a93-4616-b6b5-06ac6bd71a3a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f0161474bff54c57b95c2524d75197b5'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:50.100684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 942/1867] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:49 2022] DELETE /identity/v3/users/f0161474bff54c57b95c2524d75197b5 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:50.100898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ce5d7a25-0f50-4865-baca-1792e2e83fa1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '584031e90f424330b7c91ac7c6839859'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:50.103106 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a8e42f-62cc-4c43-a952-d6bef67ceb3e None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:50.103406 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a8e42f-62cc-4c43-a952-d6bef67ceb3e None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:50.103776 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a8e42f-62cc-4c43-a952-d6bef67ceb3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:50.113123 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ce5d7a25-0f50-4865-baca-1792e2e83fa1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '584031e90f424330b7c91ac7c6839859'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:50.116077 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ce5d7a25-0f50-4865-baca-1792e2e83fa1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '584031e90f424330b7c91ac7c6839859'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:50.131473 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 926/1868] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:49 2022] DELETE /identity/v3/users/584031e90f424330b7c91ac7c6839859 => generated 0 bytes in 404 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:26:50.136589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ae59c7-38ba-4889-a974-f442c959ddc1 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:50.139504 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ae59c7-38ba-4889-a974-f442c959ddc1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:50.140623 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ae59c7-38ba-4889-a974-f442c959ddc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:50.158778 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:50.162689 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-e6a8e42f-62cc-4c43-a952-d6bef67ceb3e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:50.164085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e6a8e42f-62cc-4c43-a952-d6bef67ceb3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:50.195324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b9ae59c7-38ba-4889-a974-f442c959ddc1 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:50.207368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b9ae59c7-38ba-4889-a974-f442c959ddc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:50.313648 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 943/1869] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:50.333707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a8e42f-62cc-4c43-a952-d6bef67ceb3e None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:50.335197 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e6a8e42f-62cc-4c43-a952-d6bef67ceb3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:50.375328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 927/1870] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:50.375328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ae59c7-38ba-4889-a974-f442c959ddc1 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:50.375328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b9ae59c7-38ba-4889-a974-f442c959ddc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:50.391145 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ae59c7-38ba-4889-a974-f442c959ddc1 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:50.396164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2b6f7e-7e43-4770-b909-a7f9d57b3ccf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:50.396836 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2b6f7e-7e43-4770-b909-a7f9d57b3ccf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:50.397156 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2b6f7e-7e43-4770-b909-a7f9d57b3ccf admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:50.397434 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2b6f7e-7e43-4770-b909-a7f9d57b3ccf admin admin] PATH_INFO: `/v3/users/e1f89af210e1447aab194bf6bc61cc32` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:50.398593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2b6f7e-7e43-4770-b909-a7f9d57b3ccf admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1f89af210e1447aab194bf6bc61cc32)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:50.402261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2b6f7e-7e43-4770-b909-a7f9d57b3ccf admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:50.478283 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a8e42f-62cc-4c43-a952-d6bef67ceb3e None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:50.482951 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f513b6-1f0f-450d-b121-b3bc96236f40 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:50.484134 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f513b6-1f0f-450d-b121-b3bc96236f40 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:50.484503 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f513b6-1f0f-450d-b121-b3bc96236f40 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:50.485146 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f513b6-1f0f-450d-b121-b3bc96236f40 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:50.509538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f2b6f7e-7e43-4770-b909-a7f9d57b3ccf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1f89af210e1447aab194bf6bc61cc32'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:50.509751 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f513b6-1f0f-450d-b121-b3bc96236f40 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:50.517442 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f513b6-1f0f-450d-b121-b3bc96236f40 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:50.533091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f2b6f7e-7e43-4770-b909-a7f9d57b3ccf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1f89af210e1447aab194bf6bc61cc32'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:50.540702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 944/1871] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:26:50.542026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f513b6-1f0f-450d-b121-b3bc96236f40 service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:50.543145 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f2b6f7e-7e43-4770-b909-a7f9d57b3ccf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1f89af210e1447aab194bf6bc61cc32'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:50.553530 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f2b6f7e-7e43-4770-b909-a7f9d57b3ccf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1f89af210e1447aab194bf6bc61cc32'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:50.553813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f513b6-1f0f-450d-b121-b3bc96236f40 service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:50.554931 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f2b6f7e-7e43-4770-b909-a7f9d57b3ccf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1f89af210e1447aab194bf6bc61cc32'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:50.558648 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ee8943-d089-48c4-8882-f8c0c0d8e2d7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:50.559273 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ee8943-d089-48c4-8882-f8c0c0d8e2d7 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:50.559559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ee8943-d089-48c4-8882-f8c0c0d8e2d7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:50.559831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ee8943-d089-48c4-8882-f8c0c0d8e2d7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:50.566080 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-33ee8943-d089-48c4-8882-f8c0c0d8e2d7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:50.567041 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 928/1872] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:50 2022] DELETE /identity/v3/users/e1f89af210e1447aab194bf6bc61cc32 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:50.571312 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887661e5-8e0b-43f6-92f1-7f170130a4dc None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:50.571715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887661e5-8e0b-43f6-92f1-7f170130a4dc None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:50.572463 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887661e5-8e0b-43f6-92f1-7f170130a4dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:50.585478 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ee8943-d089-48c4-8882-f8c0c0d8e2d7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:50.587227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ee8943-d089-48c4-8882-f8c0c0d8e2d7 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:50.604050 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 945/1873] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:50.621098 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-887661e5-8e0b-43f6-92f1-7f170130a4dc None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:50.623680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-887661e5-8e0b-43f6-92f1-7f170130a4dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:50.746668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 929/1874] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:26:50.767729 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ee8943-d089-48c4-8882-f8c0c0d8e2d7 service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:50.770076 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-33ee8943-d089-48c4-8882-f8c0c0d8e2d7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:50.785310 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ee8943-d089-48c4-8882-f8c0c0d8e2d7 service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:50.821409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:50.851492 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed22c54-f66c-4b2e-8a62-88bde669fb1e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:50.852071 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed22c54-f66c-4b2e-8a62-88bde669fb1e service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:50.852498 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed22c54-f66c-4b2e-8a62-88bde669fb1e service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:50.853093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed22c54-f66c-4b2e-8a62-88bde669fb1e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:50.880273 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed22c54-f66c-4b2e-8a62-88bde669fb1e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:50.883456 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed22c54-f66c-4b2e-8a62-88bde669fb1e service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:50.902823 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 946/1875] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:51.223416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887661e5-8e0b-43f6-92f1-7f170130a4dc None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:51.227459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-887661e5-8e0b-43f6-92f1-7f170130a4dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:51.248419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887661e5-8e0b-43f6-92f1-7f170130a4dc None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:51.256890 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf93822-fd67-4806-a89d-c02ea9e72577 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:51.261374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf93822-fd67-4806-a89d-c02ea9e72577 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:51.261374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf93822-fd67-4806-a89d-c02ea9e72577 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:51.261571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf93822-fd67-4806-a89d-c02ea9e72577 admin admin] PATH_INFO: `/v3/projects/6ec4e0fecacd44fd9e7a5b5db410ddbb` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:51.262883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf93822-fd67-4806-a89d-c02ea9e72577 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6ec4e0fecacd44fd9e7a5b5db410ddbb)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:51.265653 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf93822-fd67-4806-a89d-c02ea9e72577 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:51.276005 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed22c54-f66c-4b2e-8a62-88bde669fb1e service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:51.277527 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-bed22c54-f66c-4b2e-8a62-88bde669fb1e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:51.306463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed22c54-f66c-4b2e-8a62-88bde669fb1e service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:51.315439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43aba271-dafc-4b0b-8c68-3e3ee62e6547 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:51.316170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aba271-dafc-4b0b-8c68-3e3ee62e6547 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:51.316474 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aba271-dafc-4b0b-8c68-3e3ee62e6547 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:51.316734 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aba271-dafc-4b0b-8c68-3e3ee62e6547 admin admin] PATH_INFO: `/v3/projects/06943f9f571c4d3caa60cba6516e18a2` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:51.317385 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:51.318689 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43aba271-dafc-4b0b-8c68-3e3ee62e6547 admin admin] RBAC: Authorizing `identity:delete_project(project_id=06943f9f571c4d3caa60cba6516e18a2)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:51.320615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43aba271-dafc-4b0b-8c68-3e3ee62e6547 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:51.390722 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-adf93822-fd67-4806-a89d-c02ea9e72577 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6ec4e0fecacd44fd9e7a5b5db410ddbb', 'request_id': 'req-adf93822-fd67-4806-a89d-c02ea9e72577'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.398834 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-adf93822-fd67-4806-a89d-c02ea9e72577 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6ec4e0fecacd44fd9e7a5b5db410ddbb', 'request_id': 'req-adf93822-fd67-4806-a89d-c02ea9e72577'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.415194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-adf93822-fd67-4806-a89d-c02ea9e72577 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6ec4e0fecacd44fd9e7a5b5db410ddbb', 'request_id': 'req-adf93822-fd67-4806-a89d-c02ea9e72577'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.420746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-43aba271-dafc-4b0b-8c68-3e3ee62e6547 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '06943f9f571c4d3caa60cba6516e18a2', 'request_id': 'req-43aba271-dafc-4b0b-8c68-3e3ee62e6547'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.433715 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-43aba271-dafc-4b0b-8c68-3e3ee62e6547 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '06943f9f571c4d3caa60cba6516e18a2', 'request_id': 'req-43aba271-dafc-4b0b-8c68-3e3ee62e6547'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.447827 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-43aba271-dafc-4b0b-8c68-3e3ee62e6547 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '06943f9f571c4d3caa60cba6516e18a2', 'request_id': 'req-43aba271-dafc-4b0b-8c68-3e3ee62e6547'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.452732 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-adf93822-fd67-4806-a89d-c02ea9e72577 admin admin] The token cache is being invalidate because project 6ec4e0fecacd44fd9e7a5b5db410ddbb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:51.452898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-adf93822-fd67-4806-a89d-c02ea9e72577 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.453636 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-adf93822-fd67-4806-a89d-c02ea9e72577 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.459082 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 930/1876] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:51 2022] DELETE /identity/v3/projects/6ec4e0fecacd44fd9e7a5b5db410ddbb => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:51.459938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf93822-fd67-4806-a89d-c02ea9e72577 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:51.461388 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-adf93822-fd67-4806-a89d-c02ea9e72577 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:51.466083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-43aba271-dafc-4b0b-8c68-3e3ee62e6547 admin admin] The token cache is being invalidate because project 06943f9f571c4d3caa60cba6516e18a2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:51.466856 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-43aba271-dafc-4b0b-8c68-3e3ee62e6547 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.468090 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-43aba271-dafc-4b0b-8c68-3e3ee62e6547 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.472159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 947/1877] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:51 2022] DELETE /identity/v3/projects/06943f9f571c4d3caa60cba6516e18a2 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:51.472772 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43aba271-dafc-4b0b-8c68-3e3ee62e6547 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:51.478238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-43aba271-dafc-4b0b-8c68-3e3ee62e6547 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:51.568361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf93822-fd67-4806-a89d-c02ea9e72577 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:51.574223 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9603f68a-e892-49ab-b613-5584f3e26543 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:51.575028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9603f68a-e892-49ab-b613-5584f3e26543 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:51.575314 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9603f68a-e892-49ab-b613-5584f3e26543 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:51.575595 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9603f68a-e892-49ab-b613-5584f3e26543 admin admin] PATH_INFO: `/v3/projects/4d13d71a5ebf4d0f966b20637fb085b8` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:51.575720 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43aba271-dafc-4b0b-8c68-3e3ee62e6547 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:51.576829 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9603f68a-e892-49ab-b613-5584f3e26543 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4d13d71a5ebf4d0f966b20637fb085b8)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:51.580024 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9603f68a-e892-49ab-b613-5584f3e26543 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:51.581428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8833c3b3-6b68-4c23-989b-fa390d0f748c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:51.582526 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8833c3b3-6b68-4c23-989b-fa390d0f748c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:51.582812 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8833c3b3-6b68-4c23-989b-fa390d0f748c admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:51.583085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8833c3b3-6b68-4c23-989b-fa390d0f748c admin admin] PATH_INFO: `/v3/users/b95cf705ee0841a5b07ad7b5424cf722` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:51.584265 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8833c3b3-6b68-4c23-989b-fa390d0f748c admin admin] RBAC: Authorizing `identity:delete_user(user_id=b95cf705ee0841a5b07ad7b5424cf722)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:51.586607 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8833c3b3-6b68-4c23-989b-fa390d0f748c admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:51.652146 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-9603f68a-e892-49ab-b613-5584f3e26543 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4d13d71a5ebf4d0f966b20637fb085b8', 'request_id': 'req-9603f68a-e892-49ab-b613-5584f3e26543'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.661456 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-9603f68a-e892-49ab-b613-5584f3e26543 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4d13d71a5ebf4d0f966b20637fb085b8', 'request_id': 'req-9603f68a-e892-49ab-b613-5584f3e26543'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.667739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-8833c3b3-6b68-4c23-989b-fa390d0f748c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b95cf705ee0841a5b07ad7b5424cf722'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.670125 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-9603f68a-e892-49ab-b613-5584f3e26543 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4d13d71a5ebf4d0f966b20637fb085b8', 'request_id': 'req-9603f68a-e892-49ab-b613-5584f3e26543'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.681806 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-8833c3b3-6b68-4c23-989b-fa390d0f748c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b95cf705ee0841a5b07ad7b5424cf722'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.682661 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-8833c3b3-6b68-4c23-989b-fa390d0f748c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b95cf705ee0841a5b07ad7b5424cf722'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.683449 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-9603f68a-e892-49ab-b613-5584f3e26543 admin admin] The token cache is being invalidate because project 4d13d71a5ebf4d0f966b20637fb085b8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:51.683741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-9603f68a-e892-49ab-b613-5584f3e26543 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.684515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-9603f68a-e892-49ab-b613-5584f3e26543 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.686286 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 931/1878] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:51 2022] DELETE /identity/v3/projects/4d13d71a5ebf4d0f966b20637fb085b8 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:51.686834 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9603f68a-e892-49ab-b613-5584f3e26543 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:51.688154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9603f68a-e892-49ab-b613-5584f3e26543 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:51.692267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:51.709701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-8833c3b3-6b68-4c23-989b-fa390d0f748c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b95cf705ee0841a5b07ad7b5424cf722'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.710444 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-8833c3b3-6b68-4c23-989b-fa390d0f748c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b95cf705ee0841a5b07ad7b5424cf722'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.732650 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 948/1879] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:51 2022] DELETE /identity/v3/users/b95cf705ee0841a5b07ad7b5424cf722 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:51.736411 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:51.736535 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8833c3b3-6b68-4c23-989b-fa390d0f748c admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:51.736535 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8833c3b3-6b68-4c23-989b-fa390d0f748c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:51.776128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9603f68a-e892-49ab-b613-5584f3e26543 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:51.816576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8833c3b3-6b68-4c23-989b-fa390d0f748c admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:51.821772 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9414be8b-92d6-4480-8bbd-d5e7902e12fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:51.822716 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9414be8b-92d6-4480-8bbd-d5e7902e12fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:51.823156 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9414be8b-92d6-4480-8bbd-d5e7902e12fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:51.823601 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9414be8b-92d6-4480-8bbd-d5e7902e12fb admin admin] PATH_INFO: `/v3/users/b4e2103dd9014e76a76f268a278edad0` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:51.824743 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9414be8b-92d6-4480-8bbd-d5e7902e12fb admin admin] RBAC: Authorizing `identity:delete_user(user_id=b4e2103dd9014e76a76f268a278edad0)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:51.826898 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9414be8b-92d6-4480-8bbd-d5e7902e12fb admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:51.841911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71828a20-041a-4845-bd5e-90362c3ec7cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:51.842684 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71828a20-041a-4845-bd5e-90362c3ec7cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:51.843266 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71828a20-041a-4845-bd5e-90362c3ec7cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:51.843515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71828a20-041a-4845-bd5e-90362c3ec7cc admin admin] PATH_INFO: `/v3/users/acc83508c5c145bd84a33181afc9a0c2` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:51.844752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71828a20-041a-4845-bd5e-90362c3ec7cc admin admin] RBAC: Authorizing `identity:delete_user(user_id=acc83508c5c145bd84a33181afc9a0c2)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:51.847282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71828a20-041a-4845-bd5e-90362c3ec7cc admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:51.890389 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-9414be8b-92d6-4480-8bbd-d5e7902e12fb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b4e2103dd9014e76a76f268a278edad0'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.902038 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-9414be8b-92d6-4480-8bbd-d5e7902e12fb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b4e2103dd9014e76a76f268a278edad0'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.902876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-9414be8b-92d6-4480-8bbd-d5e7902e12fb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b4e2103dd9014e76a76f268a278edad0'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.903132 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-71828a20-041a-4845-bd5e-90362c3ec7cc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'acc83508c5c145bd84a33181afc9a0c2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.918736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-9414be8b-92d6-4480-8bbd-d5e7902e12fb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b4e2103dd9014e76a76f268a278edad0'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.919920 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-9414be8b-92d6-4480-8bbd-d5e7902e12fb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b4e2103dd9014e76a76f268a278edad0'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.921384 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-71828a20-041a-4845-bd5e-90362c3ec7cc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'acc83508c5c145bd84a33181afc9a0c2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.929367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-71828a20-041a-4845-bd5e-90362c3ec7cc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'acc83508c5c145bd84a33181afc9a0c2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.935063 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 949/1880] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:51 2022] DELETE /identity/v3/users/b4e2103dd9014e76a76f268a278edad0 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:51.959796 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-71828a20-041a-4845-bd5e-90362c3ec7cc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'acc83508c5c145bd84a33181afc9a0c2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.960603 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-71828a20-041a-4845-bd5e-90362c3ec7cc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'acc83508c5c145bd84a33181afc9a0c2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:51.968655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 932/1881] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:51 2022] DELETE /identity/v3/users/acc83508c5c145bd84a33181afc9a0c2 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:52.349420 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9414be8b-92d6-4480-8bbd-d5e7902e12fb admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:52.350894 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-9414be8b-92d6-4480-8bbd-d5e7902e12fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:52.426852 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71828a20-041a-4845-bd5e-90362c3ec7cc admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:52.428299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-71828a20-041a-4845-bd5e-90362c3ec7cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:52.445109 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:52.450169 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9414be8b-92d6-4480-8bbd-d5e7902e12fb admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:52.508784 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:52.517063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71828a20-041a-4845-bd5e-90362c3ec7cc admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:52.523800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548aa0b2-ddef-404c-9999-0fd02fd9e2d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:52.524582 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548aa0b2-ddef-404c-9999-0fd02fd9e2d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:52.524930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548aa0b2-ddef-404c-9999-0fd02fd9e2d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:52.525309 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548aa0b2-ddef-404c-9999-0fd02fd9e2d7 admin admin] PATH_INFO: `/v3/projects/6062aa736043470b8ac278988e2dee97` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:52.526554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-548aa0b2-ddef-404c-9999-0fd02fd9e2d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6062aa736043470b8ac278988e2dee97)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:52.528557 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-548aa0b2-ddef-404c-9999-0fd02fd9e2d7 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:52.535805 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da929d17-9fac-4e08-990f-fa639ed6c7cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:52.536447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da929d17-9fac-4e08-990f-fa639ed6c7cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:52.536729 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da929d17-9fac-4e08-990f-fa639ed6c7cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:52.537031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da929d17-9fac-4e08-990f-fa639ed6c7cc admin admin] PATH_INFO: `/v3/projects/dc2d44e62d6d4e508863ce69a39ca0c4` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:52.538205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da929d17-9fac-4e08-990f-fa639ed6c7cc admin admin] RBAC: Authorizing `identity:delete_project(project_id=dc2d44e62d6d4e508863ce69a39ca0c4)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:52.540199 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da929d17-9fac-4e08-990f-fa639ed6c7cc admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:52.578307 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-548aa0b2-ddef-404c-9999-0fd02fd9e2d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6062aa736043470b8ac278988e2dee97', 'request_id': 'req-548aa0b2-ddef-404c-9999-0fd02fd9e2d7'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:52.584800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-548aa0b2-ddef-404c-9999-0fd02fd9e2d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6062aa736043470b8ac278988e2dee97', 'request_id': 'req-548aa0b2-ddef-404c-9999-0fd02fd9e2d7'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:52.590007 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-da929d17-9fac-4e08-990f-fa639ed6c7cc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dc2d44e62d6d4e508863ce69a39ca0c4', 'request_id': 'req-da929d17-9fac-4e08-990f-fa639ed6c7cc'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:52.592883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-548aa0b2-ddef-404c-9999-0fd02fd9e2d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6062aa736043470b8ac278988e2dee97', 'request_id': 'req-548aa0b2-ddef-404c-9999-0fd02fd9e2d7'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:52.595519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-da929d17-9fac-4e08-990f-fa639ed6c7cc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dc2d44e62d6d4e508863ce69a39ca0c4', 'request_id': 'req-da929d17-9fac-4e08-990f-fa639ed6c7cc'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:52.602982 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-da929d17-9fac-4e08-990f-fa639ed6c7cc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dc2d44e62d6d4e508863ce69a39ca0c4', 'request_id': 'req-da929d17-9fac-4e08-990f-fa639ed6c7cc'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:52.605426 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-548aa0b2-ddef-404c-9999-0fd02fd9e2d7 admin admin] The token cache is being invalidate because project 6062aa736043470b8ac278988e2dee97 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:52.605701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-548aa0b2-ddef-404c-9999-0fd02fd9e2d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:52.606350 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-548aa0b2-ddef-404c-9999-0fd02fd9e2d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:52.611378 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 933/1882] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:52 2022] DELETE /identity/v3/projects/6062aa736043470b8ac278988e2dee97 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:26:52.611378 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548aa0b2-ddef-404c-9999-0fd02fd9e2d7 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:52.611378 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-548aa0b2-ddef-404c-9999-0fd02fd9e2d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:52.618036 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-da929d17-9fac-4e08-990f-fa639ed6c7cc admin admin] The token cache is being invalidate because project dc2d44e62d6d4e508863ce69a39ca0c4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:52.618329 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-da929d17-9fac-4e08-990f-fa639ed6c7cc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:52.618974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-da929d17-9fac-4e08-990f-fa639ed6c7cc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:52.620731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 950/1883] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:52 2022] DELETE /identity/v3/projects/dc2d44e62d6d4e508863ce69a39ca0c4 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:52.621936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ab07b7-eec2-4cf8-b9fa-edb923d6885f None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:52.622216 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ab07b7-eec2-4cf8-b9fa-edb923d6885f None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:52.622487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ab07b7-eec2-4cf8-b9fa-edb923d6885f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:52.642134 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-92ab07b7-eec2-4cf8-b9fa-edb923d6885f None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:52.645024 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-92ab07b7-eec2-4cf8-b9fa-edb923d6885f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:52.686229 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548aa0b2-ddef-404c-9999-0fd02fd9e2d7 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:52.725279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 951/1884] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:52.726587 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6500df10-d803-46d0-b180-e1625c4ca96b None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:52.726873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6500df10-d803-46d0-b180-e1625c4ca96b None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:52.727149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6500df10-d803-46d0-b180-e1625c4ca96b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:52.747634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-6500df10-d803-46d0-b180-e1625c4ca96b None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:52.748857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-6500df10-d803-46d0-b180-e1625c4ca96b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:52.757735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 952/1885] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:52.759083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1d6360-6dc0-4a99-b367-4b1bbc8abba6 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:52.759295 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1d6360-6dc0-4a99-b367-4b1bbc8abba6 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:52.759614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1d6360-6dc0-4a99-b367-4b1bbc8abba6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:52.765757 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d390d4-f5aa-444e-9784-6cfd11bcd3c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:52.766416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d390d4-f5aa-444e-9784-6cfd11bcd3c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:52.766697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d390d4-f5aa-444e-9784-6cfd11bcd3c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:52.766970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d390d4-f5aa-444e-9784-6cfd11bcd3c2 admin admin] PATH_INFO: `/v3/projects/40bf5a94be374c9e8bbba3918c8b0c7d` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:52.768154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d390d4-f5aa-444e-9784-6cfd11bcd3c2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=40bf5a94be374c9e8bbba3918c8b0c7d)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:52.770123 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d390d4-f5aa-444e-9784-6cfd11bcd3c2 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:52.779917 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-8d1d6360-6dc0-4a99-b367-4b1bbc8abba6 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:52.781170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8d1d6360-6dc0-4a99-b367-4b1bbc8abba6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:52.789252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 953/1886] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:52.790633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e968d2-e727-4368-a50e-e7b638e71f20 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:52.791084 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e968d2-e727-4368-a50e-e7b638e71f20 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:52.791497 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e968d2-e727-4368-a50e-e7b638e71f20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:52.813779 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-18e968d2-e727-4368-a50e-e7b638e71f20 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:52.815707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-18e968d2-e727-4368-a50e-e7b638e71f20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:52.817477 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:52.828862 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-37d390d4-f5aa-444e-9784-6cfd11bcd3c2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '40bf5a94be374c9e8bbba3918c8b0c7d', 'request_id': 'req-37d390d4-f5aa-444e-9784-6cfd11bcd3c2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:52.834581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-37d390d4-f5aa-444e-9784-6cfd11bcd3c2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '40bf5a94be374c9e8bbba3918c8b0c7d', 'request_id': 'req-37d390d4-f5aa-444e-9784-6cfd11bcd3c2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:52.836043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:52.843563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-37d390d4-f5aa-444e-9784-6cfd11bcd3c2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '40bf5a94be374c9e8bbba3918c8b0c7d', 'request_id': 'req-37d390d4-f5aa-444e-9784-6cfd11bcd3c2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:52.855762 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-37d390d4-f5aa-444e-9784-6cfd11bcd3c2 admin admin] The token cache is being invalidate because project 40bf5a94be374c9e8bbba3918c8b0c7d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:52.856136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-37d390d4-f5aa-444e-9784-6cfd11bcd3c2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:52.856859 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-37d390d4-f5aa-444e-9784-6cfd11bcd3c2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:52.858592 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 934/1887] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:52 2022] DELETE /identity/v3/projects/40bf5a94be374c9e8bbba3918c8b0c7d => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:52.859377 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d390d4-f5aa-444e-9784-6cfd11bcd3c2 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:52.860678 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-37d390d4-f5aa-444e-9784-6cfd11bcd3c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:52.890763 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 954/1888] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:52.895374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd0f220-a0b0-42e4-8919-f4b5398d7e4b None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:52.895666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd0f220-a0b0-42e4-8919-f4b5398d7e4b None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:52.895941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd0f220-a0b0-42e4-8919-f4b5398d7e4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:52.916141 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-7cd0f220-a0b0-42e4-8919-f4b5398d7e4b None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:52.917582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-7cd0f220-a0b0-42e4-8919-f4b5398d7e4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:52.926905 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 955/1889] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:52.927516 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd0f220-a0b0-42e4-8919-f4b5398d7e4b None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:52.939093 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d390d4-f5aa-444e-9784-6cfd11bcd3c2 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:52.940536 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd0f220-a0b0-42e4-8919-f4b5398d7e4b None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:52.945193 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fd4467-86ff-4261-a93a-def3f5619c19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:52.945833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fd4467-86ff-4261-a93a-def3f5619c19 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:52.946403 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b6ae1b-fdeb-4268-82a9-23120bd8ac9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:52.946546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fd4467-86ff-4261-a93a-def3f5619c19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:52.946632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fd4467-86ff-4261-a93a-def3f5619c19 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:52.947088 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b6ae1b-fdeb-4268-82a9-23120bd8ac9b admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:52.947371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b6ae1b-fdeb-4268-82a9-23120bd8ac9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:52.947467 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fd4467-86ff-4261-a93a-def3f5619c19 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:52.947641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b6ae1b-fdeb-4268-82a9-23120bd8ac9b admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:52.947735 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fd4467-86ff-4261-a93a-def3f5619c19 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:52.948297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b6ae1b-fdeb-4268-82a9-23120bd8ac9b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:52.948602 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b6ae1b-fdeb-4268-82a9-23120bd8ac9b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:52.949336 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fd4467-86ff-4261-a93a-def3f5619c19 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:52.950228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b6ae1b-fdeb-4268-82a9-23120bd8ac9b admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:52.961751 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 935/1890] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:26:52.963867 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 956/1891] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:26:52.966182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b6ae1b-fdeb-4268-82a9-23120bd8ac9b admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:52.966680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904adab6-3a6c-4eba-8b05-c32a90a6df36 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:52.967110 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904adab6-3a6c-4eba-8b05-c32a90a6df36 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:52.967701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904adab6-3a6c-4eba-8b05-c32a90a6df36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:52.977303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b6ae1b-fdeb-4268-82a9-23120bd8ac9b admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:52.981936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ed98da-0215-4224-b053-86381fb515c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:52.982532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ed98da-0215-4224-b053-86381fb515c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:52.982814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ed98da-0215-4224-b053-86381fb515c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:52.983091 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ed98da-0215-4224-b053-86381fb515c7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:52.983738 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ed98da-0215-4224-b053-86381fb515c7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:52.985680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ed98da-0215-4224-b053-86381fb515c7 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:52.988689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-904adab6-3a6c-4eba-8b05-c32a90a6df36 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:52.990086 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-904adab6-3a6c-4eba-8b05-c32a90a6df36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:52.999013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 936/1892] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:52.999810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904adab6-3a6c-4eba-8b05-c32a90a6df36 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.009622 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904adab6-3a6c-4eba-8b05-c32a90a6df36 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:53.014460 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038f57d7-92a7-4236-a7e9-43ad1b4b5380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:53.015055 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038f57d7-92a7-4236-a7e9-43ad1b4b5380 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.015338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038f57d7-92a7-4236-a7e9-43ad1b4b5380 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.015612 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038f57d7-92a7-4236-a7e9-43ad1b4b5380 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.016271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038f57d7-92a7-4236-a7e9-43ad1b4b5380 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:53.020336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 957/1893] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:52 2022] POST /identity/v3/projects => generated 387 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:53.020336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ed98da-0215-4224-b053-86381fb515c7 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.020570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038f57d7-92a7-4236-a7e9-43ad1b4b5380 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:53.050727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 937/1894] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:52 2022] POST /identity/v3/projects => generated 389 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:53.050727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038f57d7-92a7-4236-a7e9-43ad1b4b5380 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.091348 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ed98da-0215-4224-b053-86381fb515c7 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:53.128883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038f57d7-92a7-4236-a7e9-43ad1b4b5380 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:53.133759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0448f507-c03b-4265-a4d9-14e668a552a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:53.134459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0448f507-c03b-4265-a4d9-14e668a552a4 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.134735 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0448f507-c03b-4265-a4d9-14e668a552a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.135006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0448f507-c03b-4265-a4d9-14e668a552a4 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.135642 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0448f507-c03b-4265-a4d9-14e668a552a4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:53.137584 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0448f507-c03b-4265-a4d9-14e668a552a4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:53.164109 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 938/1895] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:53 2022] POST /identity/v3/users => generated 294 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:53.164890 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0448f507-c03b-4265-a4d9-14e668a552a4 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.176095 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0448f507-c03b-4265-a4d9-14e668a552a4 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:53.178216 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3d17b2-c919-4cd1-bcf1-85a4d66f0731 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:53.179002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3d17b2-c919-4cd1-bcf1-85a4d66f0731 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.179351 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3d17b2-c919-4cd1-bcf1-85a4d66f0731 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.179625 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3d17b2-c919-4cd1-bcf1-85a4d66f0731 admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.180277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3d17b2-c919-4cd1-bcf1-85a4d66f0731 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:53.180766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3d17b2-c919-4cd1-bcf1-85a4d66f0731 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:53.181685 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f284caff-ec3f-4f63-8081-9b556e88590f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:53.182449 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f284caff-ec3f-4f63-8081-9b556e88590f admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.182660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3d17b2-c919-4cd1-bcf1-85a4d66f0731 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:53.182864 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f284caff-ec3f-4f63-8081-9b556e88590f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.183133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f284caff-ec3f-4f63-8081-9b556e88590f admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.183809 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f284caff-ec3f-4f63-8081-9b556e88590f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:53.185801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f284caff-ec3f-4f63-8081-9b556e88590f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:53.201253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 958/1896] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:26:53.201253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3d17b2-c919-4cd1-bcf1-85a4d66f0731 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.212392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3d17b2-c919-4cd1-bcf1-85a4d66f0731 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:53.215220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 939/1897] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:53 2022] POST /identity/v3/users => generated 295 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:53.218039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f284caff-ec3f-4f63-8081-9b556e88590f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.220510 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf1f451-037d-4a0d-b3cc-1ed281e5fb49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:53.221772 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf1f451-037d-4a0d-b3cc-1ed281e5fb49 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.222061 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf1f451-037d-4a0d-b3cc-1ed281e5fb49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.222336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf1f451-037d-4a0d-b3cc-1ed281e5fb49 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.223032 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf1f451-037d-4a0d-b3cc-1ed281e5fb49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:53.223843 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=88,keystone:DELETE=4,keystone:INSERT=8 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:53.226559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf1f451-037d-4a0d-b3cc-1ed281e5fb49 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:53.229615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f284caff-ec3f-4f63-8081-9b556e88590f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:53.236046 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f017f7-feed-40ab-8a12-62525186d386 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:53.236661 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f017f7-feed-40ab-8a12-62525186d386 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.237574 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f017f7-feed-40ab-8a12-62525186d386 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.237804 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f017f7-feed-40ab-8a12-62525186d386 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.238762 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f017f7-feed-40ab-8a12-62525186d386 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:53.241204 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f017f7-feed-40ab-8a12-62525186d386 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:53.246800 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 959/1898] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:53 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:53.246800 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf1f451-037d-4a0d-b3cc-1ed281e5fb49 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.258833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf1f451-037d-4a0d-b3cc-1ed281e5fb49 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:53.270547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633f8015-f593-438d-bb70-d117f6dc48ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:53.271701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633f8015-f593-438d-bb70-d117f6dc48ff admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.271701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633f8015-f593-438d-bb70-d117f6dc48ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.272097 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633f8015-f593-438d-bb70-d117f6dc48ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.272756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633f8015-f593-438d-bb70-d117f6dc48ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:53.274588 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633f8015-f593-438d-bb70-d117f6dc48ff admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:53.279186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 940/1899] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:53 2022] POST /identity/v3/projects => generated 377 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:53.279186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f017f7-feed-40ab-8a12-62525186d386 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.280787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:53.296664 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 960/1900] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:53 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:53.297561 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633f8015-f593-438d-bb70-d117f6dc48ff admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.309733 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633f8015-f593-438d-bb70-d117f6dc48ff admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:53.352159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f017f7-feed-40ab-8a12-62525186d386 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:53.358584 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcb19ee-72c2-41d7-bfba-a3e1b760d54b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:53.359621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcb19ee-72c2-41d7-bfba-a3e1b760d54b admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.359914 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcb19ee-72c2-41d7-bfba-a3e1b760d54b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.360189 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcb19ee-72c2-41d7-bfba-a3e1b760d54b admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.361022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbcb19ee-72c2-41d7-bfba-a3e1b760d54b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:53.363887 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbcb19ee-72c2-41d7-bfba-a3e1b760d54b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:53.383246 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 941/1901] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:53 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:53.384133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcb19ee-72c2-41d7-bfba-a3e1b760d54b admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.392683 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4378f1c-8491-4d8d-851e-375499a40d41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:53.393342 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4378f1c-8491-4d8d-851e-375499a40d41 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.393622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4378f1c-8491-4d8d-851e-375499a40d41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.393900 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4378f1c-8491-4d8d-851e-375499a40d41 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.396059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4378f1c-8491-4d8d-851e-375499a40d41 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:53.396788 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4378f1c-8491-4d8d-851e-375499a40d41 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:53.402958 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcb19ee-72c2-41d7-bfba-a3e1b760d54b admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:53.412801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249a54fa-f7fb-428f-ac84-256d67f26429 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:53.413494 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249a54fa-f7fb-428f-ac84-256d67f26429 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.413871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249a54fa-f7fb-428f-ac84-256d67f26429 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.414344 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249a54fa-f7fb-428f-ac84-256d67f26429 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.415119 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249a54fa-f7fb-428f-ac84-256d67f26429 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:53.417383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249a54fa-f7fb-428f-ac84-256d67f26429 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:53.421703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 961/1902] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:53 2022] POST /identity/v3/users => generated 289 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:53.422515 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4378f1c-8491-4d8d-851e-375499a40d41 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.435868 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 942/1903] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:53 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:53.435868 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249a54fa-f7fb-428f-ac84-256d67f26429 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.436058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4378f1c-8491-4d8d-851e-375499a40d41 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:53.443382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23f8ae3-6106-49aa-ac50-973e6aa6eb75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:53.443382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23f8ae3-6106-49aa-ac50-973e6aa6eb75 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.443382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23f8ae3-6106-49aa-ac50-973e6aa6eb75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.443382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23f8ae3-6106-49aa-ac50-973e6aa6eb75 admin admin] PATH_INFO: `/v3/projects/b5e2d9838e6f4161a4f6a05d6cab3f3a/users/cb01ad6db79947a79a14f1884ccb98cd/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.444857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249a54fa-f7fb-428f-ac84-256d67f26429 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:53.449619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16762546-6540-42fd-94ed-93c2c475ca5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:53.450210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16762546-6540-42fd-94ed-93c2c475ca5b admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.450488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16762546-6540-42fd-94ed-93c2c475ca5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.451178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16762546-6540-42fd-94ed-93c2c475ca5b admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.451826 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16762546-6540-42fd-94ed-93c2c475ca5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:53.453466 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16762546-6540-42fd-94ed-93c2c475ca5b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:53.456949 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23f8ae3-6106-49aa-ac50-973e6aa6eb75 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5e2d9838e6f4161a4f6a05d6cab3f3a, user_id=cb01ad6db79947a79a14f1884ccb98cd, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:53.459645 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23f8ae3-6106-49aa-ac50-973e6aa6eb75 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:53.464328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 943/1904] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:53 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:53.465332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16762546-6540-42fd-94ed-93c2c475ca5b admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.473928 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 962/1905] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:53 2022] PUT /identity/v3/projects/b5e2d9838e6f4161a4f6a05d6cab3f3a/users/cb01ad6db79947a79a14f1884ccb98cd/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:53.474922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23f8ae3-6106-49aa-ac50-973e6aa6eb75 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.475880 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16762546-6540-42fd-94ed-93c2c475ca5b admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:53.534228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee4a3f1-ed49-48f6-a9f5-ebba96538a16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:53.534897 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee4a3f1-ed49-48f6-a9f5-ebba96538a16 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.535182 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee4a3f1-ed49-48f6-a9f5-ebba96538a16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.535455 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee4a3f1-ed49-48f6-a9f5-ebba96538a16 admin admin] PATH_INFO: `/v3/projects/d45357d9ce5445c7b02875b541c96e39/users/3a56fe9485d34844b6130d5b8e6948f5/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.543226 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23f8ae3-6106-49aa-ac50-973e6aa6eb75 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:53.551337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2afc28-ab82-41bb-977d-63c11cc8d6a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:53.552149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2afc28-ab82-41bb-977d-63c11cc8d6a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.552432 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2afc28-ab82-41bb-977d-63c11cc8d6a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.552709 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2afc28-ab82-41bb-977d-63c11cc8d6a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.553434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef2afc28-ab82-41bb-977d-63c11cc8d6a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:53.554028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee4a3f1-ed49-48f6-a9f5-ebba96538a16 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d45357d9ce5445c7b02875b541c96e39, user_id=3a56fe9485d34844b6130d5b8e6948f5, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:53.555056 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef2afc28-ab82-41bb-977d-63c11cc8d6a0 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:53.556754 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee4a3f1-ed49-48f6-a9f5-ebba96538a16 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:53.568755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 963/1906] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:53 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:53.573547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2039fc-7544-48af-acc0-483ca1b73577 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.573879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2039fc-7544-48af-acc0-483ca1b73577 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.574094 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2039fc-7544-48af-acc0-483ca1b73577 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.575059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 944/1907] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:53 2022] PUT /identity/v3/projects/d45357d9ce5445c7b02875b541c96e39/users/3a56fe9485d34844b6130d5b8e6948f5/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:53.576073 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee4a3f1-ed49-48f6-a9f5-ebba96538a16 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.606515 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-ba2039fc-7544-48af-acc0-483ca1b73577 None None] MFA Rules not processed for user `cb01ad6db79947a79a14f1884ccb98cd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:53.607831 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ba2039fc-7544-48af-acc0-483ca1b73577 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:53.632514 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:53.662890 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee4a3f1-ed49-48f6-a9f5-ebba96538a16 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:53.667722 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8609a1c-e9a5-4909-aa2a-dce833e1b326 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:53.668482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8609a1c-e9a5-4909-aa2a-dce833e1b326 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.668857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8609a1c-e9a5-4909-aa2a-dce833e1b326 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.669541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8609a1c-e9a5-4909-aa2a-dce833e1b326 admin admin] PATH_INFO: `/v3/projects/5d4f545ca0644ba1b50375b65ae4386f/users/6c32e00ed4744dd392f47094a1d5ceb5/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.674799 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:53.687360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8609a1c-e9a5-4909-aa2a-dce833e1b326 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d4f545ca0644ba1b50375b65ae4386f, user_id=6c32e00ed4744dd392f47094a1d5ceb5, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:53.690878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8609a1c-e9a5-4909-aa2a-dce833e1b326 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:53.698630 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 964/1908] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:53 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:53.703529 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e82ce0b-d0b6-4e74-aab2-b6ec307e7b04 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.703816 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e82ce0b-d0b6-4e74-aab2-b6ec307e7b04 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.704085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e82ce0b-d0b6-4e74-aab2-b6ec307e7b04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.706313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 945/1909] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:53 2022] PUT /identity/v3/projects/5d4f545ca0644ba1b50375b65ae4386f/users/6c32e00ed4744dd392f47094a1d5ceb5/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:53.707168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8609a1c-e9a5-4909-aa2a-dce833e1b326 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.737628 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-0e82ce0b-d0b6-4e74-aab2-b6ec307e7b04 None None] MFA Rules not processed for user `3a56fe9485d34844b6130d5b8e6948f5`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:53.738851 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-0e82ce0b-d0b6-4e74-aab2-b6ec307e7b04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:53.773048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8609a1c-e9a5-4909-aa2a-dce833e1b326 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:53.778174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9504409c-5d98-47de-9388-3f96558d0e5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:53.778819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9504409c-5d98-47de-9388-3f96558d0e5e admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.779094 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9504409c-5d98-47de-9388-3f96558d0e5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.779368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9504409c-5d98-47de-9388-3f96558d0e5e admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.780006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9504409c-5d98-47de-9388-3f96558d0e5e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:53.781962 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9504409c-5d98-47de-9388-3f96558d0e5e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:53.803923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 965/1910] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:53 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:53.804971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8d3f84-8728-4ff0-9a9f-0239d04909f7 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.805278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8d3f84-8728-4ff0-9a9f-0239d04909f7 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.805560 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8d3f84-8728-4ff0-9a9f-0239d04909f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.810158 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 946/1911] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:53 2022] POST /identity/v3/projects => generated 387 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:53.810964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9504409c-5d98-47de-9388-3f96558d0e5e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.833919 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-0a8d3f84-8728-4ff0-9a9f-0239d04909f7 None None] MFA Rules not processed for user `6c32e00ed4744dd392f47094a1d5ceb5`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:53.835141 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-0a8d3f84-8728-4ff0-9a9f-0239d04909f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:53.886591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9504409c-5d98-47de-9388-3f96558d0e5e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:53.892177 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c799a5b-d74d-4bfd-8c07-bbaa9dd25743 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:53.892797 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c799a5b-d74d-4bfd-8c07-bbaa9dd25743 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.893145 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c799a5b-d74d-4bfd-8c07-bbaa9dd25743 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.893447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c799a5b-d74d-4bfd-8c07-bbaa9dd25743 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.894087 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c799a5b-d74d-4bfd-8c07-bbaa9dd25743 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:53.896010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c799a5b-d74d-4bfd-8c07-bbaa9dd25743 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:53.911282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 966/1912] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:53 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:53.911282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8d3f84-8728-4ff0-9a9f-0239d04909f7 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.924019 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8d3f84-8728-4ff0-9a9f-0239d04909f7 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:53.929563 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e902c205-0939-450c-948b-b3256013e8f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:53.930862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e902c205-0939-450c-948b-b3256013e8f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:53.930862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e902c205-0939-450c-948b-b3256013e8f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:53.930862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e902c205-0939-450c-948b-b3256013e8f0 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:53.931128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 947/1913] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:53 2022] POST /identity/v3/projects => generated 387 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:53.931371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e902c205-0939-450c-948b-b3256013e8f0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:53.933456 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e902c205-0939-450c-948b-b3256013e8f0 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:53.934815 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c799a5b-d74d-4bfd-8c07-bbaa9dd25743 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.959723 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 967/1914] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:53 2022] POST /identity/v3/users => generated 294 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:53.961535 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e902c205-0939-450c-948b-b3256013e8f0 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:53.973343 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e902c205-0939-450c-948b-b3256013e8f0 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:53.975504 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:54.009822 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c799a5b-d74d-4bfd-8c07-bbaa9dd25743 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:54.015017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2522aafe-aca4-48fb-ab76-be261ffa90b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:54.015720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2522aafe-aca4-48fb-ab76-be261ffa90b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:54.016248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2522aafe-aca4-48fb-ab76-be261ffa90b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:54.016462 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2522aafe-aca4-48fb-ab76-be261ffa90b3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:54.017200 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2522aafe-aca4-48fb-ab76-be261ffa90b3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:54.019166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2522aafe-aca4-48fb-ab76-be261ffa90b3 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:54.040758 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:54.052603 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 948/1915] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:53 2022] POST /identity/v3/projects => generated 379 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:54.052603 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2522aafe-aca4-48fb-ab76-be261ffa90b3 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:54.052855 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d91ad3-7408-4b02-bc29-69a9b9017cc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:54.053402 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d91ad3-7408-4b02-bc29-69a9b9017cc5 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:54.053754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d91ad3-7408-4b02-bc29-69a9b9017cc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:54.054092 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d91ad3-7408-4b02-bc29-69a9b9017cc5 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:54.054795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d91ad3-7408-4b02-bc29-69a9b9017cc5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:54.056799 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d91ad3-7408-4b02-bc29-69a9b9017cc5 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:54.079543 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 968/1916] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:53 2022] POST /identity/v3/users => generated 300 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:54.080416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d91ad3-7408-4b02-bc29-69a9b9017cc5 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:54.091046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d91ad3-7408-4b02-bc29-69a9b9017cc5 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:54.115391 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2522aafe-aca4-48fb-ab76-be261ffa90b3 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:54.120753 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e3afa0-dc8d-4181-bf2b-10fcfbdabacb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:54.121404 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e3afa0-dc8d-4181-bf2b-10fcfbdabacb admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:54.121687 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e3afa0-dc8d-4181-bf2b-10fcfbdabacb admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:54.121965 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e3afa0-dc8d-4181-bf2b-10fcfbdabacb admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:54.122620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e3afa0-dc8d-4181-bf2b-10fcfbdabacb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:54.124233 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e3afa0-dc8d-4181-bf2b-10fcfbdabacb admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:54.135096 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 949/1917] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:54 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:54.135926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e3afa0-dc8d-4181-bf2b-10fcfbdabacb admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:54.146663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e3afa0-dc8d-4181-bf2b-10fcfbdabacb admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:54.151532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4036d06e-9662-42b9-856a-49698c3f5e71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:54.152151 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4036d06e-9662-42b9-856a-49698c3f5e71 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:54.152440 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4036d06e-9662-42b9-856a-49698c3f5e71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:54.152786 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4036d06e-9662-42b9-856a-49698c3f5e71 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:54.153497 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4036d06e-9662-42b9-856a-49698c3f5e71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:54.153859 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c538be78-ba2e-4d3c-a19b-2e30c8d999ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:54.154596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c538be78-ba2e-4d3c-a19b-2e30c8d999ed admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:54.154876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c538be78-ba2e-4d3c-a19b-2e30c8d999ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:54.155148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c538be78-ba2e-4d3c-a19b-2e30c8d999ed admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:54.155243 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4036d06e-9662-42b9-856a-49698c3f5e71 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:54.155773 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c538be78-ba2e-4d3c-a19b-2e30c8d999ed admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:54.157715 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c538be78-ba2e-4d3c-a19b-2e30c8d999ed admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:54.166590 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 950/1918] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:54 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:54.167397 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4036d06e-9662-42b9-856a-49698c3f5e71 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:54.178175 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4036d06e-9662-42b9-856a-49698c3f5e71 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:54.182948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5644efec-9289-4a1e-8c2d-248a54af7188 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:54.183544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5644efec-9289-4a1e-8c2d-248a54af7188 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:54.183825 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5644efec-9289-4a1e-8c2d-248a54af7188 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:54.184099 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5644efec-9289-4a1e-8c2d-248a54af7188 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:54.187244 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 969/1919] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:54 2022] POST /identity/v3/users => generated 290 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:54.187403 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5644efec-9289-4a1e-8c2d-248a54af7188 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:54.187506 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c538be78-ba2e-4d3c-a19b-2e30c8d999ed admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:54.187615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5644efec-9289-4a1e-8c2d-248a54af7188 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:54.195288 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c538be78-ba2e-4d3c-a19b-2e30c8d999ed admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:54.199748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 951/1920] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:54 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:54.199748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5644efec-9289-4a1e-8c2d-248a54af7188 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:54.202567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da146e3-2880-4706-ad1e-d4457a6260d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:54.203218 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da146e3-2880-4706-ad1e-d4457a6260d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:54.203496 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da146e3-2880-4706-ad1e-d4457a6260d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:54.203815 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da146e3-2880-4706-ad1e-d4457a6260d9 admin admin] PATH_INFO: `/v3/projects/64de45960b5f4f23b6b382a608ee3698/users/e08241ee3e454987a3d5049bd6d90231/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:54.209779 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5644efec-9289-4a1e-8c2d-248a54af7188 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:54.214501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f3b6ba-0d7c-423d-b83c-6a335a67ff4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:54.215088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f3b6ba-0d7c-423d-b83c-6a335a67ff4d admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:54.215366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f3b6ba-0d7c-423d-b83c-6a335a67ff4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:54.215637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f3b6ba-0d7c-423d-b83c-6a335a67ff4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:54.216276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f3b6ba-0d7c-423d-b83c-6a335a67ff4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:54.217893 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f3b6ba-0d7c-423d-b83c-6a335a67ff4d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:54.221112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da146e3-2880-4706-ad1e-d4457a6260d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=64de45960b5f4f23b6b382a608ee3698, user_id=e08241ee3e454987a3d5049bd6d90231, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:54.224472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da146e3-2880-4706-ad1e-d4457a6260d9 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:54.228641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 952/1921] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:54 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:54.229465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f3b6ba-0d7c-423d-b83c-6a335a67ff4d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:54.238799 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 970/1922] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:54 2022] PUT /identity/v3/projects/64de45960b5f4f23b6b382a608ee3698/users/e08241ee3e454987a3d5049bd6d90231/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:54.239501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f3b6ba-0d7c-423d-b83c-6a335a67ff4d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:54.240155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da146e3-2880-4706-ad1e-d4457a6260d9 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:54.297643 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e628c416-7ab0-4667-8e52-315f0e7aef46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:54.298309 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e628c416-7ab0-4667-8e52-315f0e7aef46 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:54.298595 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e628c416-7ab0-4667-8e52-315f0e7aef46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:54.298875 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e628c416-7ab0-4667-8e52-315f0e7aef46 admin admin] PATH_INFO: `/v3/projects/03fb623eee534c9bb86ec43fde16eed9/users/b332ab4eac3e46a791671861ab9bec80/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:54.304304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da146e3-2880-4706-ad1e-d4457a6260d9 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:54.309328 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d69f83-1ce7-4af0-be19-a0783e8e648a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:54.309934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d69f83-1ce7-4af0-be19-a0783e8e648a admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:54.310215 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d69f83-1ce7-4af0-be19-a0783e8e648a admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:54.310488 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d69f83-1ce7-4af0-be19-a0783e8e648a admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:54.311132 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d69f83-1ce7-4af0-be19-a0783e8e648a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:54.312731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d69f83-1ce7-4af0-be19-a0783e8e648a admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:54.314998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e628c416-7ab0-4667-8e52-315f0e7aef46 admin admin] RBAC: Authorizing `identity:create_grant(project_id=03fb623eee534c9bb86ec43fde16eed9, user_id=b332ab4eac3e46a791671861ab9bec80, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:54.317953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e628c416-7ab0-4667-8e52-315f0e7aef46 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:54.323383 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 971/1923] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:54 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:54.324467 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d69f83-1ce7-4af0-be19-a0783e8e648a admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:54.334113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 953/1924] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:54 2022] PUT /identity/v3/projects/03fb623eee534c9bb86ec43fde16eed9/users/b332ab4eac3e46a791671861ab9bec80/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:54.335258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e628c416-7ab0-4667-8e52-315f0e7aef46 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:54.335620 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d69f83-1ce7-4af0-be19-a0783e8e648a admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:54.339170 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:54.380798 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:54.404871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8943f2a-2df5-4738-b44f-8cd4d9e96ebe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:54.405658 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8943f2a-2df5-4738-b44f-8cd4d9e96ebe admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:54.405940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8943f2a-2df5-4738-b44f-8cd4d9e96ebe admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:54.406212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8943f2a-2df5-4738-b44f-8cd4d9e96ebe admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:54.406857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8943f2a-2df5-4738-b44f-8cd4d9e96ebe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:54.410615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8943f2a-2df5-4738-b44f-8cd4d9e96ebe admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:54.423751 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 972/1925] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:54 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:54.429089 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e5fbbf-b97c-496e-b718-a38852696d86 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:54.429429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e5fbbf-b97c-496e-b718-a38852696d86 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:54.429776 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e5fbbf-b97c-496e-b718-a38852696d86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:54.448366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e628c416-7ab0-4667-8e52-315f0e7aef46 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:54.458488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c08dae-3633-4d71-a8d3-065461140832 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:54.459325 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c08dae-3633-4d71-a8d3-065461140832 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:54.459684 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c08dae-3633-4d71-a8d3-065461140832 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:54.460036 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c08dae-3633-4d71-a8d3-065461140832 admin admin] PATH_INFO: `/v3/projects/057462e65af3459b9e7cd9703542cda3/users/c43831740a604b8bb5d5a3de9417db6f/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:54.471225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-06e5fbbf-b97c-496e-b718-a38852696d86 None None] MFA Rules not processed for user `b332ab4eac3e46a791671861ab9bec80`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:54.473257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-06e5fbbf-b97c-496e-b718-a38852696d86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:54.477824 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c08dae-3633-4d71-a8d3-065461140832 admin admin] RBAC: Authorizing `identity:create_grant(project_id=057462e65af3459b9e7cd9703542cda3, user_id=c43831740a604b8bb5d5a3de9417db6f, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:54.480995 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c08dae-3633-4d71-a8d3-065461140832 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:54.510113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 954/1926] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:54 2022] PUT /identity/v3/projects/057462e65af3459b9e7cd9703542cda3/users/c43831740a604b8bb5d5a3de9417db6f/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:54.512536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c08dae-3633-4d71-a8d3-065461140832 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:54.594257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c08dae-3633-4d71-a8d3-065461140832 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:54.600240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7582f8ae-276d-4e62-8d39-cb79f233452e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:54.601130 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7582f8ae-276d-4e62-8d39-cb79f233452e admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:54.601582 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7582f8ae-276d-4e62-8d39-cb79f233452e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:54.602019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7582f8ae-276d-4e62-8d39-cb79f233452e admin admin] PATH_INFO: `/v3/projects/64de45960b5f4f23b6b382a608ee3698/users/e08241ee3e454987a3d5049bd6d90231/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:54.604537 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7582f8ae-276d-4e62-8d39-cb79f233452e admin admin] RBAC: Authorizing `identity:create_grant(project_id=64de45960b5f4f23b6b382a608ee3698, user_id=e08241ee3e454987a3d5049bd6d90231, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:54.608611 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7582f8ae-276d-4e62-8d39-cb79f233452e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:54.613897 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 973/1927] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:54 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:54.615261 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7ed0cf-bbf4-4be3-b7ff-634c823081c7 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:54.615563 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7ed0cf-bbf4-4be3-b7ff-634c823081c7 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:54.615834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7ed0cf-bbf4-4be3-b7ff-634c823081c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:54.622216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 955/1928] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:54 2022] PUT /identity/v3/projects/64de45960b5f4f23b6b382a608ee3698/users/e08241ee3e454987a3d5049bd6d90231/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:54.623317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7582f8ae-276d-4e62-8d39-cb79f233452e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:54.664394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-0c7ed0cf-bbf4-4be3-b7ff-634c823081c7 None None] MFA Rules not processed for user `c43831740a604b8bb5d5a3de9417db6f`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:54.666731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-0c7ed0cf-bbf4-4be3-b7ff-634c823081c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:54.734655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7582f8ae-276d-4e62-8d39-cb79f233452e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:54.755980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255a8c1b-4db4-44b0-a699-3a51e0d10c95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:54.757264 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255a8c1b-4db4-44b0-a699-3a51e0d10c95 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:54.757717 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255a8c1b-4db4-44b0-a699-3a51e0d10c95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:54.758227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255a8c1b-4db4-44b0-a699-3a51e0d10c95 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:54.759034 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255a8c1b-4db4-44b0-a699-3a51e0d10c95 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:54.761201 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255a8c1b-4db4-44b0-a699-3a51e0d10c95 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:54.786282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 974/1929] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:54 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:54.786282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86540ae-0d19-4781-9dd1-2c729d74bfcd None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:54.786282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86540ae-0d19-4781-9dd1-2c729d74bfcd None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:54.786282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86540ae-0d19-4781-9dd1-2c729d74bfcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:54.801225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 956/1930] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:54 2022] POST /identity/v3/projects => generated 385 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:54.801225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255a8c1b-4db4-44b0-a699-3a51e0d10c95 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:54.807411 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:54.821019 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-b86540ae-0d19-4781-9dd1-2c729d74bfcd None None] MFA Rules not processed for user `e08241ee3e454987a3d5049bd6d90231`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:54.822293 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b86540ae-0d19-4781-9dd1-2c729d74bfcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:54.880241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:54.882821 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255a8c1b-4db4-44b0-a699-3a51e0d10c95 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:54.889508 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14881618-259b-4530-a6d8-e1e3131d7a7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:54.890145 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14881618-259b-4530-a6d8-e1e3131d7a7a admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:54.890433 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14881618-259b-4530-a6d8-e1e3131d7a7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:54.890772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14881618-259b-4530-a6d8-e1e3131d7a7a admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:54.891481 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14881618-259b-4530-a6d8-e1e3131d7a7a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:54.893703 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14881618-259b-4530-a6d8-e1e3131d7a7a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:54.906537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 975/1931] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:54 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:54.907336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86540ae-0d19-4781-9dd1-2c729d74bfcd None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:54.918602 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86540ae-0d19-4781-9dd1-2c729d74bfcd None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:54.923828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d4c24a-5ec2-4f2b-9f53-4679ea83c58e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:54.924956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d4c24a-5ec2-4f2b-9f53-4679ea83c58e admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:54.925258 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d4c24a-5ec2-4f2b-9f53-4679ea83c58e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:54.925593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d4c24a-5ec2-4f2b-9f53-4679ea83c58e admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:54.926289 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d4c24a-5ec2-4f2b-9f53-4679ea83c58e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:54.928329 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d4c24a-5ec2-4f2b-9f53-4679ea83c58e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:54.945185 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 957/1932] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:54 2022] POST /identity/v3/projects => generated 379 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:54.947812 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14881618-259b-4530-a6d8-e1e3131d7a7a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:54.950066 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-14881618-259b-4530-a6d8-e1e3131d7a7a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:54.957124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 976/1933] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:54 2022] POST /identity/v3/users => generated 299 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:54.957744 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d4c24a-5ec2-4f2b-9f53-4679ea83c58e admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:54.969694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d4c24a-5ec2-4f2b-9f53-4679ea83c58e admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:55.018498 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14881618-259b-4530-a6d8-e1e3131d7a7a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:55.025156 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbf0788-60d4-44e0-b724-a24bbf44c02e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:55.026118 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbf0788-60d4-44e0-b724-a24bbf44c02e service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:55.026576 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbf0788-60d4-44e0-b724-a24bbf44c02e service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:55.027038 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbf0788-60d4-44e0-b724-a24bbf44c02e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:55.038862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4fc8ea-1b76-47f2-88be-6b5ee98ec7f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:55.039491 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4fc8ea-1b76-47f2-88be-6b5ee98ec7f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:55.039898 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4fc8ea-1b76-47f2-88be-6b5ee98ec7f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:55.040173 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4fc8ea-1b76-47f2-88be-6b5ee98ec7f5 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:55.040828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f4fc8ea-1b76-47f2-88be-6b5ee98ec7f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:55.043908 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f4fc8ea-1b76-47f2-88be-6b5ee98ec7f5 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:55.074099 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 977/1934] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:54 2022] POST /identity/v3/users => generated 296 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:55.075031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4fc8ea-1b76-47f2-88be-6b5ee98ec7f5 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:55.089861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4fc8ea-1b76-47f2-88be-6b5ee98ec7f5 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:55.097631 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac2944e-7b7e-4edd-83f8-821a61e652d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:55.098256 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac2944e-7b7e-4edd-83f8-821a61e652d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:55.098531 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac2944e-7b7e-4edd-83f8-821a61e652d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:55.098799 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac2944e-7b7e-4edd-83f8-821a61e652d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:55.099450 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac2944e-7b7e-4edd-83f8-821a61e652d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:55.101101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac2944e-7b7e-4edd-83f8-821a61e652d3 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:55.115110 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 978/1935] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:55 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:55.120210 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac2944e-7b7e-4edd-83f8-821a61e652d3 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:55.135318 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac2944e-7b7e-4edd-83f8-821a61e652d3 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:55.142106 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1dcdd0b-9c44-4a37-9490-5b6695a98ac7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:55.142865 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dcdd0b-9c44-4a37-9490-5b6695a98ac7 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:55.143672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dcdd0b-9c44-4a37-9490-5b6695a98ac7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:55.143927 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dcdd0b-9c44-4a37-9490-5b6695a98ac7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:55.144620 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1dcdd0b-9c44-4a37-9490-5b6695a98ac7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:55.146291 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1dcdd0b-9c44-4a37-9490-5b6695a98ac7 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:55.153975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bbf0788-60d4-44e0-b724-a24bbf44c02e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:55.155878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bbf0788-60d4-44e0-b724-a24bbf44c02e service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:55.157515 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 979/1936] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:55 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:55.162995 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1dcdd0b-9c44-4a37-9490-5b6695a98ac7 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:55.168532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 958/1937] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:54 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:55.169394 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbf0788-60d4-44e0-b724-a24bbf44c02e service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:55.173682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1dcdd0b-9c44-4a37-9490-5b6695a98ac7 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:55.180017 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f555b880-3c7f-4b4c-8744-a8ee65d51714 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:55.181212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f555b880-3c7f-4b4c-8744-a8ee65d51714 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:55.181699 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f555b880-3c7f-4b4c-8744-a8ee65d51714 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:55.182230 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f555b880-3c7f-4b4c-8744-a8ee65d51714 admin admin] PATH_INFO: `/v3/projects/1e76228c6087450bae92877889c8d58c/users/e395aff2c86d46eca9d061a1658145df/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:55.182897 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbf0788-60d4-44e0-b724-a24bbf44c02e service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:55.187665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13172dae-489b-4441-829a-416901027d3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:55.188391 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13172dae-489b-4441-829a-416901027d3f admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:55.188680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13172dae-489b-4441-829a-416901027d3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:55.188961 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13172dae-489b-4441-829a-416901027d3f admin admin] PATH_INFO: `/v3/projects/860f76d4cfde4e3ca3d1cff9f728d622/users/6a91ed131ebd4f58b73c9a46649fef95/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:55.202830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f555b880-3c7f-4b4c-8744-a8ee65d51714 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e76228c6087450bae92877889c8d58c, user_id=e395aff2c86d46eca9d061a1658145df, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:55.211354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f555b880-3c7f-4b4c-8744-a8ee65d51714 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:55.211555 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13172dae-489b-4441-829a-416901027d3f admin admin] RBAC: Authorizing `identity:create_grant(project_id=860f76d4cfde4e3ca3d1cff9f728d622, user_id=6a91ed131ebd4f58b73c9a46649fef95, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:55.213782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13172dae-489b-4441-829a-416901027d3f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:55.228046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 980/1938] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:55 2022] PUT /identity/v3/projects/1e76228c6087450bae92877889c8d58c/users/e395aff2c86d46eca9d061a1658145df/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:55.231976 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f555b880-3c7f-4b4c-8744-a8ee65d51714 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:55.233868 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 959/1939] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:55 2022] PUT /identity/v3/projects/860f76d4cfde4e3ca3d1cff9f728d622/users/6a91ed131ebd4f58b73c9a46649fef95/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:55.238451 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13172dae-489b-4441-829a-416901027d3f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:55.273389 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:55.354988 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13172dae-489b-4441-829a-416901027d3f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:55.358088 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f555b880-3c7f-4b4c-8744-a8ee65d51714 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:55.361631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157c8193-19ae-4972-8dcd-04f38b9d933d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:55.363420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157c8193-19ae-4972-8dcd-04f38b9d933d admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:55.364450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157c8193-19ae-4972-8dcd-04f38b9d933d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:55.364884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157c8193-19ae-4972-8dcd-04f38b9d933d admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:55.365728 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157c8193-19ae-4972-8dcd-04f38b9d933d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:55.368256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157c8193-19ae-4972-8dcd-04f38b9d933d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:55.371058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb169ffc-99d3-41c4-bf99-d3ebca2f6168 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:55.371710 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb169ffc-99d3-41c4-bf99-d3ebca2f6168 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:55.372065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb169ffc-99d3-41c4-bf99-d3ebca2f6168 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:55.372433 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb169ffc-99d3-41c4-bf99-d3ebca2f6168 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:55.373679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb169ffc-99d3-41c4-bf99-d3ebca2f6168 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:55.375666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb169ffc-99d3-41c4-bf99-d3ebca2f6168 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:55.380003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 960/1940] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:55 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:55.382192 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157c8193-19ae-4972-8dcd-04f38b9d933d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:55.387664 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 981/1941] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:55 2022] GET /identity/v3/roles => generated 1420 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:26:55.388640 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb169ffc-99d3-41c4-bf99-d3ebca2f6168 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:55.394029 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157c8193-19ae-4972-8dcd-04f38b9d933d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:55.397566 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:55.398849 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87a7e5e-7ebd-4f34-9dac-7e2025d2912d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:55.399668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87a7e5e-7ebd-4f34-9dac-7e2025d2912d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:55.400270 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87a7e5e-7ebd-4f34-9dac-7e2025d2912d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:55.400748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87a7e5e-7ebd-4f34-9dac-7e2025d2912d admin admin] PATH_INFO: `/v3/users/3a56fe9485d34844b6130d5b8e6948f5` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:55.402280 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87a7e5e-7ebd-4f34-9dac-7e2025d2912d admin admin] RBAC: Authorizing `identity:delete_user(user_id=3a56fe9485d34844b6130d5b8e6948f5)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:55.404413 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87a7e5e-7ebd-4f34-9dac-7e2025d2912d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:55.407036 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb169ffc-99d3-41c4-bf99-d3ebca2f6168 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:55.418327 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8925c849-91a8-4c28-8998-0bdd40846238 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:55.419277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8925c849-91a8-4c28-8998-0bdd40846238 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:55.419819 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8925c849-91a8-4c28-8998-0bdd40846238 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:55.420259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8925c849-91a8-4c28-8998-0bdd40846238 admin admin] PATH_INFO: `/v3/projects/860f76d4cfde4e3ca3d1cff9f728d622/users/6a91ed131ebd4f58b73c9a46649fef95/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:55.422849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8925c849-91a8-4c28-8998-0bdd40846238 admin admin] RBAC: Authorizing `identity:create_grant(project_id=860f76d4cfde4e3ca3d1cff9f728d622, user_id=6a91ed131ebd4f58b73c9a46649fef95, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:55.426372 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8925c849-91a8-4c28-8998-0bdd40846238 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:55.440805 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 982/1942] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:55 2022] PUT /identity/v3/projects/860f76d4cfde4e3ca3d1cff9f728d622/users/6a91ed131ebd4f58b73c9a46649fef95/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:55.442159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8925c849-91a8-4c28-8998-0bdd40846238 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:55.476764 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d87a7e5e-7ebd-4f34-9dac-7e2025d2912d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3a56fe9485d34844b6130d5b8e6948f5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:55.491180 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d87a7e5e-7ebd-4f34-9dac-7e2025d2912d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3a56fe9485d34844b6130d5b8e6948f5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:55.499611 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d87a7e5e-7ebd-4f34-9dac-7e2025d2912d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3a56fe9485d34844b6130d5b8e6948f5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:55.510333 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d87a7e5e-7ebd-4f34-9dac-7e2025d2912d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3a56fe9485d34844b6130d5b8e6948f5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:55.511162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d87a7e5e-7ebd-4f34-9dac-7e2025d2912d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3a56fe9485d34844b6130d5b8e6948f5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:55.518464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8925c849-91a8-4c28-8998-0bdd40846238 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:55.526462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa23cf9-a864-4cf1-98a0-2f908c37e348 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:55.528538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa23cf9-a864-4cf1-98a0-2f908c37e348 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:55.528538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa23cf9-a864-4cf1-98a0-2f908c37e348 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:55.528538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa23cf9-a864-4cf1-98a0-2f908c37e348 admin admin] PATH_INFO: `/v3/projects/1e76228c6087450bae92877889c8d58c/users/e395aff2c86d46eca9d061a1658145df/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:55.531349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa23cf9-a864-4cf1-98a0-2f908c37e348 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e76228c6087450bae92877889c8d58c, user_id=e395aff2c86d46eca9d061a1658145df, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:55.545684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa23cf9-a864-4cf1-98a0-2f908c37e348 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:55.545862 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 961/1943] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:55 2022] DELETE /identity/v3/users/3a56fe9485d34844b6130d5b8e6948f5 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:55.545862 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f524cead-31e4-4d2f-85cb-aadb358dc2cf None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:55.545862 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f524cead-31e4-4d2f-85cb-aadb358dc2cf None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:55.545862 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f524cead-31e4-4d2f-85cb-aadb358dc2cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:55.550521 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 983/1944] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:55 2022] PUT /identity/v3/projects/1e76228c6087450bae92877889c8d58c/users/e395aff2c86d46eca9d061a1658145df/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:55.558641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca6b57e-a2bb-45e2-be5b-89e623a96aa9 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:55.559129 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca6b57e-a2bb-45e2-be5b-89e623a96aa9 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:55.559572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca6b57e-a2bb-45e2-be5b-89e623a96aa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:55.574536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-f524cead-31e4-4d2f-85cb-aadb358dc2cf None None] MFA Rules not processed for user `6a91ed131ebd4f58b73c9a46649fef95`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:55.576265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f524cead-31e4-4d2f-85cb-aadb358dc2cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:55.585374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-3ca6b57e-a2bb-45e2-be5b-89e623a96aa9 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:55.586701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-3ca6b57e-a2bb-45e2-be5b-89e623a96aa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:55.654551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 984/1945] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:55.655947 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09aad7b7-e6bc-434e-9146-50d8ce38a662 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:55.656301 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09aad7b7-e6bc-434e-9146-50d8ce38a662 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:55.656644 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09aad7b7-e6bc-434e-9146-50d8ce38a662 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:55.674001 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 962/1946] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:55 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:55.674001 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f524cead-31e4-4d2f-85cb-aadb358dc2cf None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:55.674001 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f524cead-31e4-4d2f-85cb-aadb358dc2cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:55.692760 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:55.692924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-09aad7b7-e6bc-434e-9146-50d8ce38a662 None None] MFA Rules not processed for user `e395aff2c86d46eca9d061a1658145df`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:55.694854 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-09aad7b7-e6bc-434e-9146-50d8ce38a662 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:55.764366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f524cead-31e4-4d2f-85cb-aadb358dc2cf None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:55.771154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce2308c-5a95-4c2a-b1dd-d2a800f2bba1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:55.772131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce2308c-5a95-4c2a-b1dd-d2a800f2bba1 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:55.772400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce2308c-5a95-4c2a-b1dd-d2a800f2bba1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:55.773314 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce2308c-5a95-4c2a-b1dd-d2a800f2bba1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:55.787105 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce2308c-5a95-4c2a-b1dd-d2a800f2bba1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:55.788570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce2308c-5a95-4c2a-b1dd-d2a800f2bba1 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:55.797087 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 985/1947] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:55 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:55.798076 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09aad7b7-e6bc-434e-9146-50d8ce38a662 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:55.803196 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-09aad7b7-e6bc-434e-9146-50d8ce38a662 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:55.808347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 963/1948] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:55.819468 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:55.820757 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce2308c-5a95-4c2a-b1dd-d2a800f2bba1 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:55.822302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9ce2308c-5a95-4c2a-b1dd-d2a800f2bba1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:55.900619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce2308c-5a95-4c2a-b1dd-d2a800f2bba1 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:55.905669 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30342e5-629c-44ba-8f6f-dc4f46a45f44 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:55.906445 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30342e5-629c-44ba-8f6f-dc4f46a45f44 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:55.906731 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30342e5-629c-44ba-8f6f-dc4f46a45f44 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:55.907005 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30342e5-629c-44ba-8f6f-dc4f46a45f44 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:55.920484 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30342e5-629c-44ba-8f6f-dc4f46a45f44 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:55.922395 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30342e5-629c-44ba-8f6f-dc4f46a45f44 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:55.928958 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09aad7b7-e6bc-434e-9146-50d8ce38a662 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:55.934787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8a46eb-41a8-4bcd-ac75-3dd4c6813e7e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:55.935504 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8a46eb-41a8-4bcd-ac75-3dd4c6813e7e service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:55.935966 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 964/1949] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:55 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:55.945389 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8a46eb-41a8-4bcd-ac75-3dd4c6813e7e service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:55.945781 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8a46eb-41a8-4bcd-ac75-3dd4c6813e7e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:55.960563 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8a46eb-41a8-4bcd-ac75-3dd4c6813e7e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:55.963726 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8a46eb-41a8-4bcd-ac75-3dd4c6813e7e service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:55.978455 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 986/1950] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:55 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:56.116457 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30342e5-629c-44ba-8f6f-dc4f46a45f44 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:56.128231 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30342e5-629c-44ba-8f6f-dc4f46a45f44 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:56.135460 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75033870-8511-4166-af3c-1cbcf2220f87 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:56.137784 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75033870-8511-4166-af3c-1cbcf2220f87 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:56.138680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75033870-8511-4166-af3c-1cbcf2220f87 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:56.139258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75033870-8511-4166-af3c-1cbcf2220f87 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:56.142245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-75033870-8511-4166-af3c-1cbcf2220f87 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:56.266598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8a46eb-41a8-4bcd-ac75-3dd4c6813e7e service cinder] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:56.268409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-5d8a46eb-41a8-4bcd-ac75-3dd4c6813e7e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:56.302255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8a46eb-41a8-4bcd-ac75-3dd4c6813e7e service cinder] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:56.315196 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75033870-8511-4166-af3c-1cbcf2220f87 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:56.316405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078a78d2-ab78-4090-b6ff-0bef223f0e4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:56.316747 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75033870-8511-4166-af3c-1cbcf2220f87 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:56.317647 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078a78d2-ab78-4090-b6ff-0bef223f0e4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:56.320156 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078a78d2-ab78-4090-b6ff-0bef223f0e4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:56.320658 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078a78d2-ab78-4090-b6ff-0bef223f0e4e admin admin] PATH_INFO: `/v3/projects/d45357d9ce5445c7b02875b541c96e39` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:56.323605 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078a78d2-ab78-4090-b6ff-0bef223f0e4e admin admin] RBAC: Authorizing `identity:delete_project(project_id=d45357d9ce5445c7b02875b541c96e39)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:56.326090 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078a78d2-ab78-4090-b6ff-0bef223f0e4e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:56.332752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 965/1951] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:56 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:56.406256 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-078a78d2-ab78-4090-b6ff-0bef223f0e4e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd45357d9ce5445c7b02875b541c96e39', 'request_id': 'req-078a78d2-ab78-4090-b6ff-0bef223f0e4e'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:56.425444 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-078a78d2-ab78-4090-b6ff-0bef223f0e4e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd45357d9ce5445c7b02875b541c96e39', 'request_id': 'req-078a78d2-ab78-4090-b6ff-0bef223f0e4e'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:56.437086 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-078a78d2-ab78-4090-b6ff-0bef223f0e4e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd45357d9ce5445c7b02875b541c96e39', 'request_id': 'req-078a78d2-ab78-4090-b6ff-0bef223f0e4e'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:56.455960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-078a78d2-ab78-4090-b6ff-0bef223f0e4e admin admin] The token cache is being invalidate because project d45357d9ce5445c7b02875b541c96e39 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:56.456305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-078a78d2-ab78-4090-b6ff-0bef223f0e4e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:56.457092 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-078a78d2-ab78-4090-b6ff-0bef223f0e4e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:56.461143 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 987/1952] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:56 2022] DELETE /identity/v3/projects/d45357d9ce5445c7b02875b541c96e39 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:26:56.464550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75033870-8511-4166-af3c-1cbcf2220f87 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:56.480513 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75033870-8511-4166-af3c-1cbcf2220f87 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:56.658220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:56.682597 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58eeedb6-7973-4966-9969-0d2d002250af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:56.684056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58eeedb6-7973-4966-9969-0d2d002250af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:56.684056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58eeedb6-7973-4966-9969-0d2d002250af admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:56.684056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58eeedb6-7973-4966-9969-0d2d002250af admin admin] PATH_INFO: `/v3/users/e08241ee3e454987a3d5049bd6d90231` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:56.686171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58eeedb6-7973-4966-9969-0d2d002250af admin admin] RBAC: Authorizing `identity:delete_user(user_id=e08241ee3e454987a3d5049bd6d90231)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:56.688347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58eeedb6-7973-4966-9969-0d2d002250af admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:56.788288 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-58eeedb6-7973-4966-9969-0d2d002250af admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e08241ee3e454987a3d5049bd6d90231'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:56.804603 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-58eeedb6-7973-4966-9969-0d2d002250af admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e08241ee3e454987a3d5049bd6d90231'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:56.813119 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-58eeedb6-7973-4966-9969-0d2d002250af admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e08241ee3e454987a3d5049bd6d90231'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:56.826106 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-58eeedb6-7973-4966-9969-0d2d002250af admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e08241ee3e454987a3d5049bd6d90231'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:56.828479 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-58eeedb6-7973-4966-9969-0d2d002250af admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e08241ee3e454987a3d5049bd6d90231'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:56.840033 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 966/1953] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:56 2022] DELETE /identity/v3/users/e08241ee3e454987a3d5049bd6d90231 => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:57.162396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078a78d2-ab78-4090-b6ff-0bef223f0e4e admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:57.164491 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-078a78d2-ab78-4090-b6ff-0bef223f0e4e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:57.176672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58eeedb6-7973-4966-9969-0d2d002250af admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:57.178155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-58eeedb6-7973-4966-9969-0d2d002250af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:57.195438 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078a78d2-ab78-4090-b6ff-0bef223f0e4e admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:57.239978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58eeedb6-7973-4966-9969-0d2d002250af admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:57.243778 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c0692a-f6cf-47df-bc76-1dcea5de371f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:57.244646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c0692a-f6cf-47df-bc76-1dcea5de371f service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:57.244927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c0692a-f6cf-47df-bc76-1dcea5de371f service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:57.245248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c0692a-f6cf-47df-bc76-1dcea5de371f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:57.247048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e5c0692a-f6cf-47df-bc76-1dcea5de371f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:57.253382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94868954-8037-4e33-a850-e8504e627c97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:57.254020 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94868954-8037-4e33-a850-e8504e627c97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:57.254379 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94868954-8037-4e33-a850-e8504e627c97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:57.254666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94868954-8037-4e33-a850-e8504e627c97 admin admin] PATH_INFO: `/v3/projects/64de45960b5f4f23b6b382a608ee3698` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:57.255780 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94868954-8037-4e33-a850-e8504e627c97 admin admin] RBAC: Authorizing `identity:delete_project(project_id=64de45960b5f4f23b6b382a608ee3698)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:57.257762 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94868954-8037-4e33-a850-e8504e627c97 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:57.311119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:57.312479 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c0692a-f6cf-47df-bc76-1dcea5de371f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:57.314299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c0692a-f6cf-47df-bc76-1dcea5de371f service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:57.321445 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-94868954-8037-4e33-a850-e8504e627c97 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '64de45960b5f4f23b6b382a608ee3698', 'request_id': 'req-94868954-8037-4e33-a850-e8504e627c97'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:57.325454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 967/1954] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:57 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:57.329718 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-94868954-8037-4e33-a850-e8504e627c97 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '64de45960b5f4f23b6b382a608ee3698', 'request_id': 'req-94868954-8037-4e33-a850-e8504e627c97'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:57.339451 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-94868954-8037-4e33-a850-e8504e627c97 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '64de45960b5f4f23b6b382a608ee3698', 'request_id': 'req-94868954-8037-4e33-a850-e8504e627c97'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:57.352126 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-94868954-8037-4e33-a850-e8504e627c97 admin admin] The token cache is being invalidate because project 64de45960b5f4f23b6b382a608ee3698 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:57.352419 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-94868954-8037-4e33-a850-e8504e627c97 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:57.353763 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-94868954-8037-4e33-a850-e8504e627c97 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:57.357085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 988/1955] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:57 2022] DELETE /identity/v3/projects/64de45960b5f4f23b6b382a608ee3698 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:57.360256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b288c9a-ec30-4a02-937b-53472344fcfe None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:57.360483 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b288c9a-ec30-4a02-937b-53472344fcfe None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:57.360799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b288c9a-ec30-4a02-937b-53472344fcfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:57.384084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3b288c9a-ec30-4a02-937b-53472344fcfe None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:57.385698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3b288c9a-ec30-4a02-937b-53472344fcfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:57.415798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:57.477213 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 968/1956] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:57.485236 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba3dada-d475-4bbb-ac2d-023fbf791389 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:57.485864 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba3dada-d475-4bbb-ac2d-023fbf791389 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:57.486146 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba3dada-d475-4bbb-ac2d-023fbf791389 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:57.509774 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-1ba3dada-d475-4bbb-ac2d-023fbf791389 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:57.510658 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-1ba3dada-d475-4bbb-ac2d-023fbf791389 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:57.523352 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 989/1957] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:26:57.524969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b288c9a-ec30-4a02-937b-53472344fcfe None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:57.535899 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b288c9a-ec30-4a02-937b-53472344fcfe None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:57.541172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2574cf99-9fe6-4bf6-8493-feb982c5cad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:57.542047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2574cf99-9fe6-4bf6-8493-feb982c5cad7 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:57.542324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2574cf99-9fe6-4bf6-8493-feb982c5cad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:57.542603 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2574cf99-9fe6-4bf6-8493-feb982c5cad7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:57.543283 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2574cf99-9fe6-4bf6-8493-feb982c5cad7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:26:57.543577 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2574cf99-9fe6-4bf6-8493-feb982c5cad7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:57.545228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2574cf99-9fe6-4bf6-8493-feb982c5cad7 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:57.564323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 969/1958] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:26:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:26:57.566180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba3dada-d475-4bbb-ac2d-023fbf791389 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:57.581778 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba3dada-d475-4bbb-ac2d-023fbf791389 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:57.586684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10341d7d-4a98-422f-bf92-5e7bf541c98e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:57.587351 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10341d7d-4a98-422f-bf92-5e7bf541c98e admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:57.587672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10341d7d-4a98-422f-bf92-5e7bf541c98e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:57.588017 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10341d7d-4a98-422f-bf92-5e7bf541c98e admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:57.588762 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10341d7d-4a98-422f-bf92-5e7bf541c98e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:57.590923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10341d7d-4a98-422f-bf92-5e7bf541c98e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:57.625128 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 990/1959] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:57 2022] POST /identity/v3/projects => generated 381 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:57.639960 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2574cf99-9fe6-4bf6-8493-feb982c5cad7 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:57.651878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2574cf99-9fe6-4bf6-8493-feb982c5cad7 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:57.729371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74488c8d-cb9f-4b4b-ac1c-147d292eed8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:57.729927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74488c8d-cb9f-4b4b-ac1c-147d292eed8a admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:57.730202 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74488c8d-cb9f-4b4b-ac1c-147d292eed8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:57.730474 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74488c8d-cb9f-4b4b-ac1c-147d292eed8a admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:57.731107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74488c8d-cb9f-4b4b-ac1c-147d292eed8a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:57.733171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74488c8d-cb9f-4b4b-ac1c-147d292eed8a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:57.761546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 970/1960] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:57 2022] POST /identity/v3/users => generated 291 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:57.761812 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10341d7d-4a98-422f-bf92-5e7bf541c98e admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:57.771807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10341d7d-4a98-422f-bf92-5e7bf541c98e admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:57.776689 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c879163-e600-4399-8cd8-bba28ff1ed81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:57.777315 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c879163-e600-4399-8cd8-bba28ff1ed81 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:57.777595 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c879163-e600-4399-8cd8-bba28ff1ed81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:57.777866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c879163-e600-4399-8cd8-bba28ff1ed81 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:57.778634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c879163-e600-4399-8cd8-bba28ff1ed81 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:57.780261 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c879163-e600-4399-8cd8-bba28ff1ed81 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:57.790206 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 991/1961] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:57.794131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74488c8d-cb9f-4b4b-ac1c-147d292eed8a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:57.803969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74488c8d-cb9f-4b4b-ac1c-147d292eed8a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:57.808518 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b2aead-495c-4e3c-80be-0706ef781538 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:57.809144 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b2aead-495c-4e3c-80be-0706ef781538 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:57.809427 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b2aead-495c-4e3c-80be-0706ef781538 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:57.809701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b2aead-495c-4e3c-80be-0706ef781538 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:57.810343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b2aead-495c-4e3c-80be-0706ef781538 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:57.811936 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b2aead-495c-4e3c-80be-0706ef781538 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:57.822855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 971/1962] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:57 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:26:57.826899 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c879163-e600-4399-8cd8-bba28ff1ed81 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:57.837356 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c879163-e600-4399-8cd8-bba28ff1ed81 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:57.841920 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4355f11b-8289-4b84-a5eb-0cc2bad2447e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:57.842563 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4355f11b-8289-4b84-a5eb-0cc2bad2447e admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:57.842844 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4355f11b-8289-4b84-a5eb-0cc2bad2447e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:57.843119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4355f11b-8289-4b84-a5eb-0cc2bad2447e admin admin] PATH_INFO: `/v3/projects/9a432ec30c1b4ef4bdcc6f5e937acb44/users/3d0f5ea96b594f04a486f8ed458c07f7/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:57.857662 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4355f11b-8289-4b84-a5eb-0cc2bad2447e admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a432ec30c1b4ef4bdcc6f5e937acb44, user_id=3d0f5ea96b594f04a486f8ed458c07f7, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:57.860375 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4355f11b-8289-4b84-a5eb-0cc2bad2447e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:57.873954 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 992/1963] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:57 2022] PUT /identity/v3/projects/9a432ec30c1b4ef4bdcc6f5e937acb44/users/3d0f5ea96b594f04a486f8ed458c07f7/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:57.886516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bff19db-c34c-48fc-911f-688799c23413 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:57.886872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bff19db-c34c-48fc-911f-688799c23413 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:57.887184 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bff19db-c34c-48fc-911f-688799c23413 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:57.916820 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-9bff19db-c34c-48fc-911f-688799c23413 None None] MFA Rules not processed for user `3d0f5ea96b594f04a486f8ed458c07f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:57.918292 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9bff19db-c34c-48fc-911f-688799c23413 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:57.982656 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 972/1964] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:57 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:57.987557 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4355f11b-8289-4b84-a5eb-0cc2bad2447e admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:58.051952 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4355f11b-8289-4b84-a5eb-0cc2bad2447e admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:58.056557 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80630ef2-edba-4950-b206-eaa8fbb89a95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:58.057191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80630ef2-edba-4950-b206-eaa8fbb89a95 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:58.057471 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80630ef2-edba-4950-b206-eaa8fbb89a95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:58.057740 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80630ef2-edba-4950-b206-eaa8fbb89a95 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:58.058376 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80630ef2-edba-4950-b206-eaa8fbb89a95 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:58.060297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80630ef2-edba-4950-b206-eaa8fbb89a95 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:58.092959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 993/1965] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:26:57 2022] POST /identity/v3/projects => generated 381 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:58.094207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bff19db-c34c-48fc-911f-688799c23413 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:58.107605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bff19db-c34c-48fc-911f-688799c23413 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:58.128700 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:58.171888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcbedb3c-ca3a-461f-8f24-a4730382385d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:58.172526 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcbedb3c-ca3a-461f-8f24-a4730382385d admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:58.172806 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcbedb3c-ca3a-461f-8f24-a4730382385d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:58.173115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcbedb3c-ca3a-461f-8f24-a4730382385d admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:58.173747 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcbedb3c-ca3a-461f-8f24-a4730382385d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:58.176212 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcbedb3c-ca3a-461f-8f24-a4730382385d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:58.207826 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 973/1966] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:26:58 2022] POST /identity/v3/users => generated 297 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:26:58.211804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80630ef2-edba-4950-b206-eaa8fbb89a95 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:58.222763 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80630ef2-edba-4950-b206-eaa8fbb89a95 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:58.227417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb635ba-b2c5-43fd-9332-41899c5c8a98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:58.228096 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb635ba-b2c5-43fd-9332-41899c5c8a98 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:58.228385 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb635ba-b2c5-43fd-9332-41899c5c8a98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:58.228660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb635ba-b2c5-43fd-9332-41899c5c8a98 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:58.229344 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb635ba-b2c5-43fd-9332-41899c5c8a98 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:58.230976 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb635ba-b2c5-43fd-9332-41899c5c8a98 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:58.242146 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 994/1967] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:58 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:26:58.246513 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcbedb3c-ca3a-461f-8f24-a4730382385d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:58.256305 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcbedb3c-ca3a-461f-8f24-a4730382385d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:58.260756 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9ce8bf-d067-4fb8-9812-b1acf80903ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:58.261437 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9ce8bf-d067-4fb8-9812-b1acf80903ad admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:58.261715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9ce8bf-d067-4fb8-9812-b1acf80903ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:58.261987 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9ce8bf-d067-4fb8-9812-b1acf80903ad admin admin] PATH_INFO: `/v3/projects/cfd6f23c48dd47249b9cb6fac8f7e667/users/83bd65b53c9747d6b073e1098c3050df/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:58.278525 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9ce8bf-d067-4fb8-9812-b1acf80903ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=cfd6f23c48dd47249b9cb6fac8f7e667, user_id=83bd65b53c9747d6b073e1098c3050df, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:58.281136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9ce8bf-d067-4fb8-9812-b1acf80903ad admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:58.294876 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 974/1968] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:58 2022] PUT /identity/v3/projects/cfd6f23c48dd47249b9cb6fac8f7e667/users/83bd65b53c9747d6b073e1098c3050df/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:58.298985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb635ba-b2c5-43fd-9332-41899c5c8a98 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:58.309943 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb635ba-b2c5-43fd-9332-41899c5c8a98 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:58.342407 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:58.370157 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49bf61fa-047a-44f8-a7ad-af773735f2f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:58.370763 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bf61fa-047a-44f8-a7ad-af773735f2f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:58.371061 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bf61fa-047a-44f8-a7ad-af773735f2f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:58.371327 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bf61fa-047a-44f8-a7ad-af773735f2f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:58.371954 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49bf61fa-047a-44f8-a7ad-af773735f2f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:58.373585 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49bf61fa-047a-44f8-a7ad-af773735f2f1 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:58.383809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 995/1969] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:26:58 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:26:58.387985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9ce8bf-d067-4fb8-9812-b1acf80903ad admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:58.398036 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9ce8bf-d067-4fb8-9812-b1acf80903ad admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:58.402541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dfc2a0b-6f03-40ac-8305-691e46bb862e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:58.403186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfc2a0b-6f03-40ac-8305-691e46bb862e admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:58.403466 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfc2a0b-6f03-40ac-8305-691e46bb862e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:58.403738 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfc2a0b-6f03-40ac-8305-691e46bb862e admin admin] PATH_INFO: `/v3/projects/cfd6f23c48dd47249b9cb6fac8f7e667/users/83bd65b53c9747d6b073e1098c3050df/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:58.405624 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dfc2a0b-6f03-40ac-8305-691e46bb862e admin admin] RBAC: Authorizing `identity:create_grant(project_id=cfd6f23c48dd47249b9cb6fac8f7e667, user_id=83bd65b53c9747d6b073e1098c3050df, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:58.408258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dfc2a0b-6f03-40ac-8305-691e46bb862e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:58.418788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 975/1970] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:26:58 2022] PUT /identity/v3/projects/cfd6f23c48dd47249b9cb6fac8f7e667/users/83bd65b53c9747d6b073e1098c3050df/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:58.430370 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f466f84-f8e0-4d16-803a-3878e388ca24 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:58.430746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f466f84-f8e0-4d16-803a-3878e388ca24 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:58.431021 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f466f84-f8e0-4d16-803a-3878e388ca24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:58.465331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-7f466f84-f8e0-4d16-803a-3878e388ca24 None None] MFA Rules not processed for user `83bd65b53c9747d6b073e1098c3050df`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:58.467170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-7f466f84-f8e0-4d16-803a-3878e388ca24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:58.541634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 996/1971] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:58 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:58.552597 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dfc2a0b-6f03-40ac-8305-691e46bb862e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:58.554421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0dfc2a0b-6f03-40ac-8305-691e46bb862e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:58.614135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dfc2a0b-6f03-40ac-8305-691e46bb862e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:58.618436 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f465fcb-127a-4144-8585-e580bb2037a8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:58.620312 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f465fcb-127a-4144-8585-e580bb2037a8 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:58.620594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f465fcb-127a-4144-8585-e580bb2037a8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:58.620862 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f465fcb-127a-4144-8585-e580bb2037a8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:58.630592 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f465fcb-127a-4144-8585-e580bb2037a8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:58.632040 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f465fcb-127a-4144-8585-e580bb2037a8 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:58.642070 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 976/1972] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:58 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:59.073371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f466f84-f8e0-4d16-803a-3878e388ca24 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:59.149989 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f466f84-f8e0-4d16-803a-3878e388ca24 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:59.163923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d3fc45-700b-4864-98ca-941846220e69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:59.164551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d3fc45-700b-4864-98ca-941846220e69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:59.164777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d3fc45-700b-4864-98ca-941846220e69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:59.165152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d3fc45-700b-4864-98ca-941846220e69 admin admin] PATH_INFO: `/v3/users/3d0f5ea96b594f04a486f8ed458c07f7` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:59.166338 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d3fc45-700b-4864-98ca-941846220e69 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3d0f5ea96b594f04a486f8ed458c07f7)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:59.168563 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d3fc45-700b-4864-98ca-941846220e69 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:59.230119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-97d3fc45-700b-4864-98ca-941846220e69 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3d0f5ea96b594f04a486f8ed458c07f7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:59.238800 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-97d3fc45-700b-4864-98ca-941846220e69 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3d0f5ea96b594f04a486f8ed458c07f7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:59.239526 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-97d3fc45-700b-4864-98ca-941846220e69 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3d0f5ea96b594f04a486f8ed458c07f7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:59.245297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:59.254137 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-97d3fc45-700b-4864-98ca-941846220e69 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3d0f5ea96b594f04a486f8ed458c07f7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:59.254774 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-97d3fc45-700b-4864-98ca-941846220e69 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3d0f5ea96b594f04a486f8ed458c07f7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:59.267006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 997/1973] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:59 2022] DELETE /identity/v3/users/3d0f5ea96b594f04a486f8ed458c07f7 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:59.271561 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27804abc-c406-44b1-a237-a852b6399f6e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:59.272013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27804abc-c406-44b1-a237-a852b6399f6e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:59.272138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27804abc-c406-44b1-a237-a852b6399f6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:59.291577 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-27804abc-c406-44b1-a237-a852b6399f6e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:26:59.292770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-27804abc-c406-44b1-a237-a852b6399f6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:59.351088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 977/1974] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:26:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:26:59.360607 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d3fc45-700b-4864-98ca-941846220e69 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:59.362105 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-97d3fc45-700b-4864-98ca-941846220e69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:59.420343 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d3fc45-700b-4864-98ca-941846220e69 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:59.425255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c4313b-b3a6-4f78-9acf-67ac94fb68be service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:59.426000 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c4313b-b3a6-4f78-9acf-67ac94fb68be service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:59.426323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c4313b-b3a6-4f78-9acf-67ac94fb68be service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:59.426640 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c4313b-b3a6-4f78-9acf-67ac94fb68be service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:59.436484 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c4313b-b3a6-4f78-9acf-67ac94fb68be service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:59.437963 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c4313b-b3a6-4f78-9acf-67ac94fb68be service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:59.447828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 998/1975] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:26:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:26:59.774313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27804abc-c406-44b1-a237-a852b6399f6e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:59.775289 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-27804abc-c406-44b1-a237-a852b6399f6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:26:59.798711 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27804abc-c406-44b1-a237-a852b6399f6e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:26:59.804073 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa25b84-9c5a-4bf9-ab14-81275003a6fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:26:59.804724 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa25b84-9c5a-4bf9-ab14-81275003a6fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:26:59.805009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa25b84-9c5a-4bf9-ab14-81275003a6fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:26:59.805328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa25b84-9c5a-4bf9-ab14-81275003a6fa admin admin] PATH_INFO: `/v3/projects/9a432ec30c1b4ef4bdcc6f5e937acb44` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:26:59.806517 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa25b84-9c5a-4bf9-ab14-81275003a6fa admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a432ec30c1b4ef4bdcc6f5e937acb44)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:26:59.808442 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa25b84-9c5a-4bf9-ab14-81275003a6fa admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:26:59.823549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:26:59.890217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5fa25b84-9c5a-4bf9-ab14-81275003a6fa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a432ec30c1b4ef4bdcc6f5e937acb44', 'request_id': 'req-5fa25b84-9c5a-4bf9-ab14-81275003a6fa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:59.896895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5fa25b84-9c5a-4bf9-ab14-81275003a6fa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a432ec30c1b4ef4bdcc6f5e937acb44', 'request_id': 'req-5fa25b84-9c5a-4bf9-ab14-81275003a6fa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:59.906263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5fa25b84-9c5a-4bf9-ab14-81275003a6fa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a432ec30c1b4ef4bdcc6f5e937acb44', 'request_id': 'req-5fa25b84-9c5a-4bf9-ab14-81275003a6fa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:59.918886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5fa25b84-9c5a-4bf9-ab14-81275003a6fa admin admin] The token cache is being invalidate because project 9a432ec30c1b4ef4bdcc6f5e937acb44 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:26:59.919179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5fa25b84-9c5a-4bf9-ab14-81275003a6fa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:59.920444 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5fa25b84-9c5a-4bf9-ab14-81275003a6fa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:26:59.922266 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 978/1976] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:26:59 2022] DELETE /identity/v3/projects/9a432ec30c1b4ef4bdcc6f5e937acb44 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:26:59.926481 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c4313b-b3a6-4f78-9acf-67ac94fb68be service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:26:59.936593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c4313b-b3a6-4f78-9acf-67ac94fb68be service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:00.008278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bde32d-3a4a-41ea-a0a5-648a01807952 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:00.008848 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bde32d-3a4a-41ea-a0a5-648a01807952 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:00.008848 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bde32d-3a4a-41ea-a0a5-648a01807952 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:00.009360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bde32d-3a4a-41ea-a0a5-648a01807952 admin admin] PATH_INFO: `/v3/users/83bd65b53c9747d6b073e1098c3050df` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:00.010922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bde32d-3a4a-41ea-a0a5-648a01807952 admin admin] RBAC: Authorizing `identity:delete_user(user_id=83bd65b53c9747d6b073e1098c3050df)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:00.013071 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bde32d-3a4a-41ea-a0a5-648a01807952 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:00.066242 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b0bde32d-3a4a-41ea-a0a5-648a01807952 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '83bd65b53c9747d6b073e1098c3050df'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:00.073791 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b0bde32d-3a4a-41ea-a0a5-648a01807952 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '83bd65b53c9747d6b073e1098c3050df'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:00.074410 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b0bde32d-3a4a-41ea-a0a5-648a01807952 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '83bd65b53c9747d6b073e1098c3050df'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:00.088673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b0bde32d-3a4a-41ea-a0a5-648a01807952 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '83bd65b53c9747d6b073e1098c3050df'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:00.089570 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b0bde32d-3a4a-41ea-a0a5-648a01807952 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '83bd65b53c9747d6b073e1098c3050df'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:00.098858 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:00.103710 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 999/1977] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:26:59 2022] DELETE /identity/v3/users/83bd65b53c9747d6b073e1098c3050df => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:00.407880 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa25b84-9c5a-4bf9-ab14-81275003a6fa admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:00.410614 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5fa25b84-9c5a-4bf9-ab14-81275003a6fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:00.427651 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa25b84-9c5a-4bf9-ab14-81275003a6fa admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:00.491716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a788ef-3b64-458a-9ba5-8f47a1a1d449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:00.492369 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a788ef-3b64-458a-9ba5-8f47a1a1d449 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:00.492650 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a788ef-3b64-458a-9ba5-8f47a1a1d449 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:00.492930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a788ef-3b64-458a-9ba5-8f47a1a1d449 admin admin] PATH_INFO: `/v3/projects/cfd6f23c48dd47249b9cb6fac8f7e667` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:00.494108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a788ef-3b64-458a-9ba5-8f47a1a1d449 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cfd6f23c48dd47249b9cb6fac8f7e667)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:00.496292 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a788ef-3b64-458a-9ba5-8f47a1a1d449 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:00.547820 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a9a788ef-3b64-458a-9ba5-8f47a1a1d449 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cfd6f23c48dd47249b9cb6fac8f7e667', 'request_id': 'req-a9a788ef-3b64-458a-9ba5-8f47a1a1d449'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:00.553990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a9a788ef-3b64-458a-9ba5-8f47a1a1d449 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cfd6f23c48dd47249b9cb6fac8f7e667', 'request_id': 'req-a9a788ef-3b64-458a-9ba5-8f47a1a1d449'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:00.562127 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a9a788ef-3b64-458a-9ba5-8f47a1a1d449 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cfd6f23c48dd47249b9cb6fac8f7e667', 'request_id': 'req-a9a788ef-3b64-458a-9ba5-8f47a1a1d449'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:00.575661 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a9a788ef-3b64-458a-9ba5-8f47a1a1d449 admin admin] The token cache is being invalidate because project cfd6f23c48dd47249b9cb6fac8f7e667 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:00.575952 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a9a788ef-3b64-458a-9ba5-8f47a1a1d449 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:00.576651 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a9a788ef-3b64-458a-9ba5-8f47a1a1d449 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:00.578669 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 979/1978] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:00 2022] DELETE /identity/v3/projects/cfd6f23c48dd47249b9cb6fac8f7e667 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:00.584588 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b5f088-05bc-471a-8c31-0ac8149d193f None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:00.585114 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b5f088-05bc-471a-8c31-0ac8149d193f None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:00.585257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b5f088-05bc-471a-8c31-0ac8149d193f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:00.620501 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-c9b5f088-05bc-471a-8c31-0ac8149d193f None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:00.622082 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c9b5f088-05bc-471a-8c31-0ac8149d193f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:00.697285 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1000/1979] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:00.704501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbac63b-b379-46d4-88d4-eed1e1459ea9 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:00.704788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbac63b-b379-46d4-88d4-eed1e1459ea9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:00.705151 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbac63b-b379-46d4-88d4-eed1e1459ea9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:00.730265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-6cbac63b-b379-46d4-88d4-eed1e1459ea9 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:00.732433 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6cbac63b-b379-46d4-88d4-eed1e1459ea9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:00.745726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 980/1980] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:00.747731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b5f088-05bc-471a-8c31-0ac8149d193f None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:00.758409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b5f088-05bc-471a-8c31-0ac8149d193f None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:00.764790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ae30fb-ba09-491f-9818-23c6aaaf8f71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:00.765565 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ae30fb-ba09-491f-9818-23c6aaaf8f71 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:00.766152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ae30fb-ba09-491f-9818-23c6aaaf8f71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:00.766263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ae30fb-ba09-491f-9818-23c6aaaf8f71 admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:00.767046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ae30fb-ba09-491f-9818-23c6aaaf8f71 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:27:00.767324 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ae30fb-ba09-491f-9818-23c6aaaf8f71 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:00.769991 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ae30fb-ba09-491f-9818-23c6aaaf8f71 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:00.797186 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1001/1981] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:27:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:27:00.800745 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbac63b-b379-46d4-88d4-eed1e1459ea9 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:00.812552 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbac63b-b379-46d4-88d4-eed1e1459ea9 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:00.826994 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62b58f3-e494-4b0e-86e1-9efb7fbf9a0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:00.831835 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62b58f3-e494-4b0e-86e1-9efb7fbf9a0c admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:00.831835 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62b58f3-e494-4b0e-86e1-9efb7fbf9a0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:00.831835 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62b58f3-e494-4b0e-86e1-9efb7fbf9a0c admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:00.840820 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b62b58f3-e494-4b0e-86e1-9efb7fbf9a0c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:00.848326 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b62b58f3-e494-4b0e-86e1-9efb7fbf9a0c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:00.909641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 981/1982] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:00 2022] POST /identity/v3/projects => generated 377 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:27:00.912529 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ae30fb-ba09-491f-9818-23c6aaaf8f71 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:00.927560 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ae30fb-ba09-491f-9818-23c6aaaf8f71 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:01.046635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d436afe-c2af-4ec1-a200-fcd94efbb2e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:01.047281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d436afe-c2af-4ec1-a200-fcd94efbb2e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:01.047523 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d436afe-c2af-4ec1-a200-fcd94efbb2e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:01.048001 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d436afe-c2af-4ec1-a200-fcd94efbb2e1 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:01.049271 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d436afe-c2af-4ec1-a200-fcd94efbb2e1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:01.052159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d436afe-c2af-4ec1-a200-fcd94efbb2e1 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:01.083088 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1002/1983] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:00 2022] POST /identity/v3/users => generated 289 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:01.087644 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62b58f3-e494-4b0e-86e1-9efb7fbf9a0c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:01.093414 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:01.101663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62b58f3-e494-4b0e-86e1-9efb7fbf9a0c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:01.108152 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8f0a2b-e36f-4ac7-9010-2088d28b16c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:01.109070 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8f0a2b-e36f-4ac7-9010-2088d28b16c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:01.109480 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8f0a2b-e36f-4ac7-9010-2088d28b16c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:01.109923 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8f0a2b-e36f-4ac7-9010-2088d28b16c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:01.110760 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8f0a2b-e36f-4ac7-9010-2088d28b16c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:01.125372 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8f0a2b-e36f-4ac7-9010-2088d28b16c1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:01.141956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 982/1984] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:01 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:01.147605 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d436afe-c2af-4ec1-a200-fcd94efbb2e1 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:01.161470 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d436afe-c2af-4ec1-a200-fcd94efbb2e1 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:01.175885 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755d2bf5-4454-4d6a-ae5d-491e8486c02c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:01.176701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755d2bf5-4454-4d6a-ae5d-491e8486c02c admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:01.177153 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755d2bf5-4454-4d6a-ae5d-491e8486c02c admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:01.177575 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755d2bf5-4454-4d6a-ae5d-491e8486c02c admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:01.178719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755d2bf5-4454-4d6a-ae5d-491e8486c02c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:01.180618 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755d2bf5-4454-4d6a-ae5d-491e8486c02c admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:01.202003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8f0a2b-e36f-4ac7-9010-2088d28b16c1 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:01.203286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1003/1985] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:01 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:01.224062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8f0a2b-e36f-4ac7-9010-2088d28b16c1 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:01.224062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d49f390-e118-4c03-a0c1-ee8c5e22f59e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:01.224062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d49f390-e118-4c03-a0c1-ee8c5e22f59e admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:01.224062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d49f390-e118-4c03-a0c1-ee8c5e22f59e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:01.224494 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d49f390-e118-4c03-a0c1-ee8c5e22f59e admin admin] PATH_INFO: `/v3/projects/8701651cc71743fbbc9ac74a7aa3845c/users/6130659864db48cf925a1745ef946098/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:01.242040 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d49f390-e118-4c03-a0c1-ee8c5e22f59e admin admin] RBAC: Authorizing `identity:create_grant(project_id=8701651cc71743fbbc9ac74a7aa3845c, user_id=6130659864db48cf925a1745ef946098, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:01.245344 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d49f390-e118-4c03-a0c1-ee8c5e22f59e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:01.265156 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 983/1986] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:01 2022] PUT /identity/v3/projects/8701651cc71743fbbc9ac74a7aa3845c/users/6130659864db48cf925a1745ef946098/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:01.277169 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5868e0cb-6b12-493c-952a-a11ceaa0ef8e None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:01.277451 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5868e0cb-6b12-493c-952a-a11ceaa0ef8e None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:01.277718 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5868e0cb-6b12-493c-952a-a11ceaa0ef8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:01.308311 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-5868e0cb-6b12-493c-952a-a11ceaa0ef8e None None] MFA Rules not processed for user `6130659864db48cf925a1745ef946098`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:01.309657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-5868e0cb-6b12-493c-952a-a11ceaa0ef8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:01.388507 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:01.405225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1004/1987] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:01 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:27:01.422102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d49f390-e118-4c03-a0c1-ee8c5e22f59e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:01.424846 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5d49f390-e118-4c03-a0c1-ee8c5e22f59e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:01.532221 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d49f390-e118-4c03-a0c1-ee8c5e22f59e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:01.536953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87d431e-46b6-4b24-939e-d8dbd2bef94e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '354b8228661c4369b3bca2e631f3f140', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:01.537731 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87d431e-46b6-4b24-939e-d8dbd2bef94e service glance] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:01.537970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87d431e-46b6-4b24-939e-d8dbd2bef94e service glance] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:01.538200 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87d431e-46b6-4b24-939e-d8dbd2bef94e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:01.547546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87d431e-46b6-4b24-939e-d8dbd2bef94e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:01.549696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87d431e-46b6-4b24-939e-d8dbd2bef94e service glance] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:01.563891 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 984/1988] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:01 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:02.910753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5868e0cb-6b12-493c-952a-a11ceaa0ef8e None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:02.912600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-5868e0cb-6b12-493c-952a-a11ceaa0ef8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:03.006832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5868e0cb-6b12-493c-952a-a11ceaa0ef8e None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:03.011443 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675dbc54-254e-4ea2-b4d9-c5df5ab68fc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:03.012298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675dbc54-254e-4ea2-b4d9-c5df5ab68fc2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:03.012634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675dbc54-254e-4ea2-b4d9-c5df5ab68fc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:03.012978 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675dbc54-254e-4ea2-b4d9-c5df5ab68fc2 admin admin] PATH_INFO: `/v3/users/6c32e00ed4744dd392f47094a1d5ceb5` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:03.014129 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675dbc54-254e-4ea2-b4d9-c5df5ab68fc2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6c32e00ed4744dd392f47094a1d5ceb5)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:03.016050 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675dbc54-254e-4ea2-b4d9-c5df5ab68fc2 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:03.073703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-675dbc54-254e-4ea2-b4d9-c5df5ab68fc2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6c32e00ed4744dd392f47094a1d5ceb5'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:03.082535 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-675dbc54-254e-4ea2-b4d9-c5df5ab68fc2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6c32e00ed4744dd392f47094a1d5ceb5'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:03.083190 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-675dbc54-254e-4ea2-b4d9-c5df5ab68fc2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6c32e00ed4744dd392f47094a1d5ceb5'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:03.098866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-675dbc54-254e-4ea2-b4d9-c5df5ab68fc2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6c32e00ed4744dd392f47094a1d5ceb5'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:03.099636 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-675dbc54-254e-4ea2-b4d9-c5df5ab68fc2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6c32e00ed4744dd392f47094a1d5ceb5'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:03.113059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444ea7cb-67b5-463f-adbd-458870ad5966 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:03.113691 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444ea7cb-67b5-463f-adbd-458870ad5966 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:03.113691 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444ea7cb-67b5-463f-adbd-458870ad5966 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:03.115391 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 985/1989] 10.222.0.85 () {54 vars in 921 bytes} [Tue Aug 30 04:27:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:27:03.123984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1005/1990] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:02 2022] DELETE /identity/v3/users/6c32e00ed4744dd392f47094a1d5ceb5 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:03.128181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbefce21-2835-4d84-9506-b85bbcc2f88d None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:03.128463 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbefce21-2835-4d84-9506-b85bbcc2f88d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:03.128766 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbefce21-2835-4d84-9506-b85bbcc2f88d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:03.133465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bff700f-0e08-42b8-becd-158434b91c43 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:03.133945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bff700f-0e08-42b8-becd-158434b91c43 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:03.134269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bff700f-0e08-42b8-becd-158434b91c43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:03.151069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-cbefce21-2835-4d84-9506-b85bbcc2f88d None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:03.152600 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cbefce21-2835-4d84-9506-b85bbcc2f88d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:03.160379 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-7bff700f-0e08-42b8-becd-158434b91c43 None None] MFA Rules not processed for user `4bc4d455ebf041d488b1199ca58358a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:03.161840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-7bff700f-0e08-42b8-becd-158434b91c43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:03.264791 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1006/1991] 10.222.0.85 () {58 vars in 1018 bytes} [Tue Aug 30 04:27:03 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:03.277593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 986/1992] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:03.287650 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bff700f-0e08-42b8-becd-158434b91c43 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:03.289170 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbefce21-2835-4d84-9506-b85bbcc2f88d None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:03.289307 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-7bff700f-0e08-42b8-becd-158434b91c43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:03.290991 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cbefce21-2835-4d84-9506-b85bbcc2f88d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:03.303613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bff700f-0e08-42b8-becd-158434b91c43 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:03.309423 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a51153-d317-451c-ba5f-7a9802236a83 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:03.310041 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a51153-d317-451c-ba5f-7a9802236a83 service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:03.310325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a51153-d317-451c-ba5f-7a9802236a83 service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:03.310978 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a51153-d317-451c-ba5f-7a9802236a83 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:03.318842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:03.321330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a51153-d317-451c-ba5f-7a9802236a83 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:03.323256 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a51153-d317-451c-ba5f-7a9802236a83 service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:03.337104 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1007/1993] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:03 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:03.364966 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbefce21-2835-4d84-9506-b85bbcc2f88d None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:03.369865 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ad24ec-0367-4b22-98ea-d12bbac524b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:03.370832 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad24ec-0367-4b22-98ea-d12bbac524b0 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:03.370832 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad24ec-0367-4b22-98ea-d12bbac524b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:03.371019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad24ec-0367-4b22-98ea-d12bbac524b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:03.377616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:03.383155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ad24ec-0367-4b22-98ea-d12bbac524b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:03.385546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ad24ec-0367-4b22-98ea-d12bbac524b0 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:03.396683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 987/1994] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:03.665687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a51153-d317-451c-ba5f-7a9802236a83 service nova] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:03.667119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c9a51153-d317-451c-ba5f-7a9802236a83 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:03.681354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a51153-d317-451c-ba5f-7a9802236a83 service nova] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:03.685920 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1a3bbf-7e5b-4042-8a52-8ef6720211e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:03.686640 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1a3bbf-7e5b-4042-8a52-8ef6720211e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:03.687487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1a3bbf-7e5b-4042-8a52-8ef6720211e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:03.687772 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1a3bbf-7e5b-4042-8a52-8ef6720211e7 admin admin] PATH_INFO: `/v3/projects/5d4f545ca0644ba1b50375b65ae4386f` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:03.689144 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1a3bbf-7e5b-4042-8a52-8ef6720211e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d4f545ca0644ba1b50375b65ae4386f)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:03.691596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1a3bbf-7e5b-4042-8a52-8ef6720211e7 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:03.740897 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-1e1a3bbf-7e5b-4042-8a52-8ef6720211e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d4f545ca0644ba1b50375b65ae4386f', 'request_id': 'req-1e1a3bbf-7e5b-4042-8a52-8ef6720211e7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:03.746858 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-1e1a3bbf-7e5b-4042-8a52-8ef6720211e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d4f545ca0644ba1b50375b65ae4386f', 'request_id': 'req-1e1a3bbf-7e5b-4042-8a52-8ef6720211e7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:03.754905 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-1e1a3bbf-7e5b-4042-8a52-8ef6720211e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d4f545ca0644ba1b50375b65ae4386f', 'request_id': 'req-1e1a3bbf-7e5b-4042-8a52-8ef6720211e7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:03.766617 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-1e1a3bbf-7e5b-4042-8a52-8ef6720211e7 admin admin] The token cache is being invalidate because project 5d4f545ca0644ba1b50375b65ae4386f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:03.766909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-1e1a3bbf-7e5b-4042-8a52-8ef6720211e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:03.767577 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-1e1a3bbf-7e5b-4042-8a52-8ef6720211e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:03.769731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1008/1995] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:03 2022] DELETE /identity/v3/projects/5d4f545ca0644ba1b50375b65ae4386f => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:03.773800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ad24ec-0367-4b22-98ea-d12bbac524b0 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:03.783595 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ad24ec-0367-4b22-98ea-d12bbac524b0 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:03.840473 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3ef092-ad84-4839-9e7c-73091e4153a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:03.841135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3ef092-ad84-4839-9e7c-73091e4153a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:03.841423 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3ef092-ad84-4839-9e7c-73091e4153a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:03.841695 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3ef092-ad84-4839-9e7c-73091e4153a7 admin admin] PATH_INFO: `/v3/users/c43831740a604b8bb5d5a3de9417db6f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:03.842798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3ef092-ad84-4839-9e7c-73091e4153a7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c43831740a604b8bb5d5a3de9417db6f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:03.844720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3ef092-ad84-4839-9e7c-73091e4153a7 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:03.890561 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8c3ef092-ad84-4839-9e7c-73091e4153a7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c43831740a604b8bb5d5a3de9417db6f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:03.902889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8c3ef092-ad84-4839-9e7c-73091e4153a7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c43831740a604b8bb5d5a3de9417db6f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:03.909718 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8c3ef092-ad84-4839-9e7c-73091e4153a7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c43831740a604b8bb5d5a3de9417db6f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:03.916944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8c3ef092-ad84-4839-9e7c-73091e4153a7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c43831740a604b8bb5d5a3de9417db6f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:03.917617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8c3ef092-ad84-4839-9e7c-73091e4153a7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c43831740a604b8bb5d5a3de9417db6f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:03.923542 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 988/1996] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:03 2022] DELETE /identity/v3/users/c43831740a604b8bb5d5a3de9417db6f => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:04.298642 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1a3bbf-7e5b-4042-8a52-8ef6720211e7 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:04.301314 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-1e1a3bbf-7e5b-4042-8a52-8ef6720211e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:04.332595 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1a3bbf-7e5b-4042-8a52-8ef6720211e7 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:04.403116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7166e4b7-44a2-4875-9d35-91051c9b4cac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:04.403855 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7166e4b7-44a2-4875-9d35-91051c9b4cac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:04.404203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7166e4b7-44a2-4875-9d35-91051c9b4cac admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:04.404537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7166e4b7-44a2-4875-9d35-91051c9b4cac admin admin] PATH_INFO: `/v3/projects/057462e65af3459b9e7cd9703542cda3` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:04.405761 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7166e4b7-44a2-4875-9d35-91051c9b4cac admin admin] RBAC: Authorizing `identity:delete_project(project_id=057462e65af3459b9e7cd9703542cda3)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:04.407784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7166e4b7-44a2-4875-9d35-91051c9b4cac admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:04.455493 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-7166e4b7-44a2-4875-9d35-91051c9b4cac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '057462e65af3459b9e7cd9703542cda3', 'request_id': 'req-7166e4b7-44a2-4875-9d35-91051c9b4cac'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:04.460867 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-7166e4b7-44a2-4875-9d35-91051c9b4cac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '057462e65af3459b9e7cd9703542cda3', 'request_id': 'req-7166e4b7-44a2-4875-9d35-91051c9b4cac'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:04.468780 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-7166e4b7-44a2-4875-9d35-91051c9b4cac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '057462e65af3459b9e7cd9703542cda3', 'request_id': 'req-7166e4b7-44a2-4875-9d35-91051c9b4cac'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:04.480568 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-7166e4b7-44a2-4875-9d35-91051c9b4cac admin admin] The token cache is being invalidate because project 057462e65af3459b9e7cd9703542cda3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:04.480855 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-7166e4b7-44a2-4875-9d35-91051c9b4cac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:04.481518 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-7166e4b7-44a2-4875-9d35-91051c9b4cac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:04.483029 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1009/1997] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:04 2022] DELETE /identity/v3/projects/057462e65af3459b9e7cd9703542cda3 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:04.487694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3ef092-ad84-4839-9e7c-73091e4153a7 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:04.489135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8c3ef092-ad84-4839-9e7c-73091e4153a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:04.502126 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3ef092-ad84-4839-9e7c-73091e4153a7 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:04.558245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d0d86c-e335-44ea-8542-c61e45e917ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:04.558791 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d0d86c-e335-44ea-8542-c61e45e917ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:04.559067 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d0d86c-e335-44ea-8542-c61e45e917ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:04.559336 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d0d86c-e335-44ea-8542-c61e45e917ac admin admin] PATH_INFO: `/v3/users/6a91ed131ebd4f58b73c9a46649fef95` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:04.560443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d0d86c-e335-44ea-8542-c61e45e917ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=6a91ed131ebd4f58b73c9a46649fef95)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:04.562367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d0d86c-e335-44ea-8542-c61e45e917ac admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:04.575647 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:04.609594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b2d0d86c-e335-44ea-8542-c61e45e917ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6a91ed131ebd4f58b73c9a46649fef95'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:04.622289 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b2d0d86c-e335-44ea-8542-c61e45e917ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6a91ed131ebd4f58b73c9a46649fef95'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:04.629226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b2d0d86c-e335-44ea-8542-c61e45e917ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6a91ed131ebd4f58b73c9a46649fef95'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:04.636915 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b2d0d86c-e335-44ea-8542-c61e45e917ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6a91ed131ebd4f58b73c9a46649fef95'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:04.637571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b2d0d86c-e335-44ea-8542-c61e45e917ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6a91ed131ebd4f58b73c9a46649fef95'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:04.643965 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 989/1998] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:04 2022] DELETE /identity/v3/users/6a91ed131ebd4f58b73c9a46649fef95 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:04.913538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7166e4b7-44a2-4875-9d35-91051c9b4cac admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:04.915137 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-7166e4b7-44a2-4875-9d35-91051c9b4cac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:04.928014 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7166e4b7-44a2-4875-9d35-91051c9b4cac admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:04.932389 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:04.990033 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382814ad-d0c7-4b9e-a511-63d209f5079f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:04.990669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382814ad-d0c7-4b9e-a511-63d209f5079f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:04.990951 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382814ad-d0c7-4b9e-a511-63d209f5079f admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:04.991226 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382814ad-d0c7-4b9e-a511-63d209f5079f admin admin] PATH_INFO: `/v3/projects/860f76d4cfde4e3ca3d1cff9f728d622` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:04.992344 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382814ad-d0c7-4b9e-a511-63d209f5079f admin admin] RBAC: Authorizing `identity:delete_project(project_id=860f76d4cfde4e3ca3d1cff9f728d622)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:04.994547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382814ad-d0c7-4b9e-a511-63d209f5079f admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:05.040824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-382814ad-d0c7-4b9e-a511-63d209f5079f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '860f76d4cfde4e3ca3d1cff9f728d622', 'request_id': 'req-382814ad-d0c7-4b9e-a511-63d209f5079f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:05.046470 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-382814ad-d0c7-4b9e-a511-63d209f5079f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '860f76d4cfde4e3ca3d1cff9f728d622', 'request_id': 'req-382814ad-d0c7-4b9e-a511-63d209f5079f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:05.053514 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-382814ad-d0c7-4b9e-a511-63d209f5079f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '860f76d4cfde4e3ca3d1cff9f728d622', 'request_id': 'req-382814ad-d0c7-4b9e-a511-63d209f5079f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:05.064383 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-382814ad-d0c7-4b9e-a511-63d209f5079f admin admin] The token cache is being invalidate because project 860f76d4cfde4e3ca3d1cff9f728d622 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:05.065211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-382814ad-d0c7-4b9e-a511-63d209f5079f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:05.065840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-382814ad-d0c7-4b9e-a511-63d209f5079f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:05.067313 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1010/1999] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:04 2022] DELETE /identity/v3/projects/860f76d4cfde4e3ca3d1cff9f728d622 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:05.073463 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ddeb39-6eac-4043-a4cb-bdfd3db46aef None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:05.073813 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ddeb39-6eac-4043-a4cb-bdfd3db46aef None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:05.074026 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ddeb39-6eac-4043-a4cb-bdfd3db46aef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:05.093790 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-64ddeb39-6eac-4043-a4cb-bdfd3db46aef None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:05.094971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-64ddeb39-6eac-4043-a4cb-bdfd3db46aef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:05.155108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 990/2000] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:05.163326 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874c4a70-dfb4-4a95-9be3-6366f9333361 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:05.163610 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874c4a70-dfb4-4a95-9be3-6366f9333361 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:05.163880 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874c4a70-dfb4-4a95-9be3-6366f9333361 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:05.183657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-874c4a70-dfb4-4a95-9be3-6366f9333361 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:05.184856 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-874c4a70-dfb4-4a95-9be3-6366f9333361 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:05.193526 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1011/2001] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:05.197365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ddeb39-6eac-4043-a4cb-bdfd3db46aef None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:05.207446 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ddeb39-6eac-4043-a4cb-bdfd3db46aef None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:05.211988 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0949a53d-3e2e-400a-917c-1c00b11898d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:05.212664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0949a53d-3e2e-400a-917c-1c00b11898d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:05.212942 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0949a53d-3e2e-400a-917c-1c00b11898d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:05.213232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0949a53d-3e2e-400a-917c-1c00b11898d7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:05.213975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0949a53d-3e2e-400a-917c-1c00b11898d7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:27:05.214268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0949a53d-3e2e-400a-917c-1c00b11898d7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:05.215871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0949a53d-3e2e-400a-917c-1c00b11898d7 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:05.228087 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 991/2002] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:27:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:27:05.232544 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874c4a70-dfb4-4a95-9be3-6366f9333361 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:05.241633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874c4a70-dfb4-4a95-9be3-6366f9333361 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:05.247058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b868336b-3d23-4a8b-aa65-b0c1d9544e2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:05.247788 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b868336b-3d23-4a8b-aa65-b0c1d9544e2e admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:05.248131 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b868336b-3d23-4a8b-aa65-b0c1d9544e2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:05.248435 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b868336b-3d23-4a8b-aa65-b0c1d9544e2e admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:05.249190 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b868336b-3d23-4a8b-aa65-b0c1d9544e2e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:05.251127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b868336b-3d23-4a8b-aa65-b0c1d9544e2e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:05.282672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1012/2003] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:05 2022] POST /identity/v3/projects => generated 385 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:05.286721 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0949a53d-3e2e-400a-917c-1c00b11898d7 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:05.298510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0949a53d-3e2e-400a-917c-1c00b11898d7 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:05.370451 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce0d94e-53b8-47c0-8944-4d6e26831df2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:05.371070 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce0d94e-53b8-47c0-8944-4d6e26831df2 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:05.371349 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce0d94e-53b8-47c0-8944-4d6e26831df2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:05.371645 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce0d94e-53b8-47c0-8944-4d6e26831df2 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:05.372276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce0d94e-53b8-47c0-8944-4d6e26831df2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:05.374246 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce0d94e-53b8-47c0-8944-4d6e26831df2 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:05.400445 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 992/2004] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:05 2022] POST /identity/v3/users => generated 293 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:27:05.404724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b868336b-3d23-4a8b-aa65-b0c1d9544e2e admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:05.416541 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b868336b-3d23-4a8b-aa65-b0c1d9544e2e admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:05.421512 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1196275c-fe34-4ffb-8fc0-b0174989ef9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:05.422115 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1196275c-fe34-4ffb-8fc0-b0174989ef9e admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:05.422391 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1196275c-fe34-4ffb-8fc0-b0174989ef9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:05.422659 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1196275c-fe34-4ffb-8fc0-b0174989ef9e admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:05.423300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1196275c-fe34-4ffb-8fc0-b0174989ef9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:05.425469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1196275c-fe34-4ffb-8fc0-b0174989ef9e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:05.436025 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1013/2005] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:05 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:05.442015 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce0d94e-53b8-47c0-8944-4d6e26831df2 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:05.457223 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce0d94e-53b8-47c0-8944-4d6e26831df2 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:05.463011 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4479ded6-5c8b-496c-ac67-43aa9f7d6c95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:05.463628 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4479ded6-5c8b-496c-ac67-43aa9f7d6c95 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:05.463906 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4479ded6-5c8b-496c-ac67-43aa9f7d6c95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:05.464183 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4479ded6-5c8b-496c-ac67-43aa9f7d6c95 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:05.464834 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4479ded6-5c8b-496c-ac67-43aa9f7d6c95 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:05.466674 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4479ded6-5c8b-496c-ac67-43aa9f7d6c95 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:05.469638 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:05.487177 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 993/2006] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:05 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:05.488614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1196275c-fe34-4ffb-8fc0-b0174989ef9e admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:05.500699 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1196275c-fe34-4ffb-8fc0-b0174989ef9e admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:05.506992 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd02d3a-f19a-49b5-97bb-80076ae9a74e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:05.507721 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd02d3a-f19a-49b5-97bb-80076ae9a74e admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:05.508180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd02d3a-f19a-49b5-97bb-80076ae9a74e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:05.508524 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd02d3a-f19a-49b5-97bb-80076ae9a74e admin admin] PATH_INFO: `/v3/projects/8d82c4611d04411999cfd12e3d6049ea/users/b3e43a4eb5a04cedae369d3a5c8918c5/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:05.524251 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd02d3a-f19a-49b5-97bb-80076ae9a74e admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d82c4611d04411999cfd12e3d6049ea, user_id=b3e43a4eb5a04cedae369d3a5c8918c5, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:05.526945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd02d3a-f19a-49b5-97bb-80076ae9a74e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:05.540079 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1014/2007] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:05 2022] PUT /identity/v3/projects/8d82c4611d04411999cfd12e3d6049ea/users/b3e43a4eb5a04cedae369d3a5c8918c5/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:05.552702 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce3894e-39c1-475b-bc44-1d065a4fd456 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:05.553233 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce3894e-39c1-475b-bc44-1d065a4fd456 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:05.553310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce3894e-39c1-475b-bc44-1d065a4fd456 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:05.584673 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5ce3894e-39c1-475b-bc44-1d065a4fd456 None None] MFA Rules not processed for user `b3e43a4eb5a04cedae369d3a5c8918c5`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:05.585868 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5ce3894e-39c1-475b-bc44-1d065a4fd456 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:05.665166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 994/2008] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:05 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:05.679819 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd02d3a-f19a-49b5-97bb-80076ae9a74e admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:05.681617 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-2fd02d3a-f19a-49b5-97bb-80076ae9a74e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:05.723027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:05.758187 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd02d3a-f19a-49b5-97bb-80076ae9a74e admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:05.761876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cebaa4-c120-49c7-ab66-82ce2e31eca4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:05.762485 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cebaa4-c120-49c7-ab66-82ce2e31eca4 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:05.762763 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cebaa4-c120-49c7-ab66-82ce2e31eca4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:05.763032 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cebaa4-c120-49c7-ab66-82ce2e31eca4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:05.771793 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cebaa4-c120-49c7-ab66-82ce2e31eca4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:05.773257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cebaa4-c120-49c7-ab66-82ce2e31eca4 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:05.782579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1015/2009] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:05 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:05.803074 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce3894e-39c1-475b-bc44-1d065a4fd456 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:05.864517 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce3894e-39c1-475b-bc44-1d065a4fd456 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:05.868831 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dc9ead-8cb2-45f1-9cb0-58692a06eeee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:05.869786 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dc9ead-8cb2-45f1-9cb0-58692a06eeee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:05.870070 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dc9ead-8cb2-45f1-9cb0-58692a06eeee admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:05.870344 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dc9ead-8cb2-45f1-9cb0-58692a06eeee admin admin] PATH_INFO: `/v3/users/b3e43a4eb5a04cedae369d3a5c8918c5` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:05.871468 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dc9ead-8cb2-45f1-9cb0-58692a06eeee admin admin] RBAC: Authorizing `identity:delete_user(user_id=b3e43a4eb5a04cedae369d3a5c8918c5)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:05.873454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dc9ead-8cb2-45f1-9cb0-58692a06eeee admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:05.928258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a2dc9ead-8cb2-45f1-9cb0-58692a06eeee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b3e43a4eb5a04cedae369d3a5c8918c5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:05.942141 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a2dc9ead-8cb2-45f1-9cb0-58692a06eeee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b3e43a4eb5a04cedae369d3a5c8918c5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:05.949757 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a2dc9ead-8cb2-45f1-9cb0-58692a06eeee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b3e43a4eb5a04cedae369d3a5c8918c5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:05.960457 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a2dc9ead-8cb2-45f1-9cb0-58692a06eeee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b3e43a4eb5a04cedae369d3a5c8918c5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:05.961216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a2dc9ead-8cb2-45f1-9cb0-58692a06eeee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b3e43a4eb5a04cedae369d3a5c8918c5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:05.969705 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 995/2010] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:05 2022] DELETE /identity/v3/users/b3e43a4eb5a04cedae369d3a5c8918c5 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:05.974338 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e516a19e-e903-4f98-8f1b-8af0ee799d22 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:05.974627 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e516a19e-e903-4f98-8f1b-8af0ee799d22 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:05.974902 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e516a19e-e903-4f98-8f1b-8af0ee799d22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:06.000140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-e516a19e-e903-4f98-8f1b-8af0ee799d22 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:06.001501 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e516a19e-e903-4f98-8f1b-8af0ee799d22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:06.083898 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1016/2011] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:06.093700 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dc9ead-8cb2-45f1-9cb0-58692a06eeee admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:06.095154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a2dc9ead-8cb2-45f1-9cb0-58692a06eeee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:06.105909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:06.162659 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dc9ead-8cb2-45f1-9cb0-58692a06eeee admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:06.166443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d8e208-1567-480c-9b29-0243c0afb1ff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:06.167059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d8e208-1567-480c-9b29-0243c0afb1ff service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:06.167338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d8e208-1567-480c-9b29-0243c0afb1ff service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:06.167612 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d8e208-1567-480c-9b29-0243c0afb1ff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:06.177633 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d8e208-1567-480c-9b29-0243c0afb1ff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:06.179073 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d8e208-1567-480c-9b29-0243c0afb1ff service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:06.188779 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 996/2012] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:06.472789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e516a19e-e903-4f98-8f1b-8af0ee799d22 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:06.476291 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e516a19e-e903-4f98-8f1b-8af0ee799d22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:06.492381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e516a19e-e903-4f98-8f1b-8af0ee799d22 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:06.498188 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10abdc3-4907-414d-b1bf-b459d6f0d292 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:06.499027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10abdc3-4907-414d-b1bf-b459d6f0d292 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:06.501253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10abdc3-4907-414d-b1bf-b459d6f0d292 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:06.501742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10abdc3-4907-414d-b1bf-b459d6f0d292 admin admin] PATH_INFO: `/v3/projects/8d82c4611d04411999cfd12e3d6049ea` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:06.503231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10abdc3-4907-414d-b1bf-b459d6f0d292 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8d82c4611d04411999cfd12e3d6049ea)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:06.505984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10abdc3-4907-414d-b1bf-b459d6f0d292 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:06.578596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d10abdc3-4907-414d-b1bf-b459d6f0d292 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8d82c4611d04411999cfd12e3d6049ea', 'request_id': 'req-d10abdc3-4907-414d-b1bf-b459d6f0d292'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:06.589074 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d10abdc3-4907-414d-b1bf-b459d6f0d292 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8d82c4611d04411999cfd12e3d6049ea', 'request_id': 'req-d10abdc3-4907-414d-b1bf-b459d6f0d292'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:06.599147 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d10abdc3-4907-414d-b1bf-b459d6f0d292 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8d82c4611d04411999cfd12e3d6049ea', 'request_id': 'req-d10abdc3-4907-414d-b1bf-b459d6f0d292'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:06.620792 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d10abdc3-4907-414d-b1bf-b459d6f0d292 admin admin] The token cache is being invalidate because project 8d82c4611d04411999cfd12e3d6049ea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:06.621213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d10abdc3-4907-414d-b1bf-b459d6f0d292 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:06.622252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d10abdc3-4907-414d-b1bf-b459d6f0d292 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:06.624232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1017/2013] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:06 2022] DELETE /identity/v3/projects/8d82c4611d04411999cfd12e3d6049ea => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:06.629661 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4daa9421-edee-47f2-8956-c6e4cc9e7932 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:06.630187 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4daa9421-edee-47f2-8956-c6e4cc9e7932 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:06.630329 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4daa9421-edee-47f2-8956-c6e4cc9e7932 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:06.652956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4daa9421-edee-47f2-8956-c6e4cc9e7932 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:06.654926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4daa9421-edee-47f2-8956-c6e4cc9e7932 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:06.725482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 997/2014] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:06.733228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b990c91d-5d40-4a02-b364-61f2d9543171 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:06.733639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b990c91d-5d40-4a02-b364-61f2d9543171 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:06.733911 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b990c91d-5d40-4a02-b364-61f2d9543171 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:06.759288 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-b990c91d-5d40-4a02-b364-61f2d9543171 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:06.760543 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b990c91d-5d40-4a02-b364-61f2d9543171 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:06.769266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1018/2015] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:06.776246 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4daa9421-edee-47f2-8956-c6e4cc9e7932 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:06.783736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4daa9421-edee-47f2-8956-c6e4cc9e7932 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:06.788550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628aeb5c-aeec-4b55-94ba-255c5fffc619 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:06.789198 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628aeb5c-aeec-4b55-94ba-255c5fffc619 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:06.789479 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628aeb5c-aeec-4b55-94ba-255c5fffc619 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:06.789850 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628aeb5c-aeec-4b55-94ba-255c5fffc619 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:06.790534 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628aeb5c-aeec-4b55-94ba-255c5fffc619 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:27:06.790797 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628aeb5c-aeec-4b55-94ba-255c5fffc619 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:06.792374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628aeb5c-aeec-4b55-94ba-255c5fffc619 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:06.806002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 998/2016] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:27:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:27:06.810442 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b990c91d-5d40-4a02-b364-61f2d9543171 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:06.820579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b990c91d-5d40-4a02-b364-61f2d9543171 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:06.832150 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e92b914-96bc-42df-a77d-e0a0a62218d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:06.832765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e92b914-96bc-42df-a77d-e0a0a62218d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:06.833068 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e92b914-96bc-42df-a77d-e0a0a62218d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:06.833429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e92b914-96bc-42df-a77d-e0a0a62218d5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:06.834123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e92b914-96bc-42df-a77d-e0a0a62218d5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:06.836228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e92b914-96bc-42df-a77d-e0a0a62218d5 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:06.872101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1019/2017] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:06 2022] POST /identity/v3/projects => generated 373 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:27:06.876184 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628aeb5c-aeec-4b55-94ba-255c5fffc619 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:06.888066 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628aeb5c-aeec-4b55-94ba-255c5fffc619 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:06.991588 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d32d6bf-a23e-4022-80bc-e92c49c95864 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:06.992257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d32d6bf-a23e-4022-80bc-e92c49c95864 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:06.992540 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d32d6bf-a23e-4022-80bc-e92c49c95864 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:06.992818 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d32d6bf-a23e-4022-80bc-e92c49c95864 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:06.993506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d32d6bf-a23e-4022-80bc-e92c49c95864 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:06.995891 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d32d6bf-a23e-4022-80bc-e92c49c95864 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:07.027743 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e92b914-96bc-42df-a77d-e0a0a62218d5 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:07.029467 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-0e92b914-96bc-42df-a77d-e0a0a62218d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:07.030768 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 999/2018] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:06 2022] POST /identity/v3/users => generated 287 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:07.035457 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d32d6bf-a23e-4022-80bc-e92c49c95864 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:07.046441 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d32d6bf-a23e-4022-80bc-e92c49c95864 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:07.056231 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e258823b-a086-4608-b50b-79486826c9e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:07.056992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e258823b-a086-4608-b50b-79486826c9e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:07.057383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e258823b-a086-4608-b50b-79486826c9e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:07.057764 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e258823b-a086-4608-b50b-79486826c9e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:07.058733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e258823b-a086-4608-b50b-79486826c9e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:07.060351 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e258823b-a086-4608-b50b-79486826c9e1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:07.067752 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:07.077736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1000/2019] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:07 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:27:07.082710 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e258823b-a086-4608-b50b-79486826c9e1 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:07.112222 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e92b914-96bc-42df-a77d-e0a0a62218d5 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:07.112365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e258823b-a086-4608-b50b-79486826c9e1 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:07.116040 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77956a5a-0253-443f-83de-994a415a98f7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:07.116778 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77956a5a-0253-443f-83de-994a415a98f7 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:07.117076 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77956a5a-0253-443f-83de-994a415a98f7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:07.117528 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77956a5a-0253-443f-83de-994a415a98f7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:07.120346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-77956a5a-0253-443f-83de-994a415a98f7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:07.124363 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7809ae32-50af-435a-8f8d-a19ea62fb151 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:07.125276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7809ae32-50af-435a-8f8d-a19ea62fb151 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:07.125661 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7809ae32-50af-435a-8f8d-a19ea62fb151 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:07.126376 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7809ae32-50af-435a-8f8d-a19ea62fb151 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:07.128201 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7809ae32-50af-435a-8f8d-a19ea62fb151 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:07.133948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7809ae32-50af-435a-8f8d-a19ea62fb151 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:07.148796 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1001/2020] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:07 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:07.149885 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7809ae32-50af-435a-8f8d-a19ea62fb151 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:07.157791 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:07.162376 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7809ae32-50af-435a-8f8d-a19ea62fb151 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:07.173178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4403e86-708b-4429-964e-5b63a6b99bf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:07.174043 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4403e86-708b-4429-964e-5b63a6b99bf8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:07.174384 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4403e86-708b-4429-964e-5b63a6b99bf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:07.174735 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4403e86-708b-4429-964e-5b63a6b99bf8 admin admin] PATH_INFO: `/v3/projects/61d37e609d474ce4a6b628612a787172/users/ffa831bf02da45a493fabf6e09d1bd8f/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:07.191360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4403e86-708b-4429-964e-5b63a6b99bf8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=61d37e609d474ce4a6b628612a787172, user_id=ffa831bf02da45a493fabf6e09d1bd8f, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:07.193550 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77956a5a-0253-443f-83de-994a415a98f7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:07.194327 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4403e86-708b-4429-964e-5b63a6b99bf8 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:07.195457 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77956a5a-0253-443f-83de-994a415a98f7 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:07.215007 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1020/2021] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:07 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:07.215699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1002/2022] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:07 2022] PUT /identity/v3/projects/61d37e609d474ce4a6b628612a787172/users/ffa831bf02da45a493fabf6e09d1bd8f/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:07.227505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9248b7df-49cf-486c-9b26-c8c46fff285c None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:07.227716 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9248b7df-49cf-486c-9b26-c8c46fff285c None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:07.228649 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9248b7df-49cf-486c-9b26-c8c46fff285c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:07.269975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-9248b7df-49cf-486c-9b26-c8c46fff285c None None] MFA Rules not processed for user `ffa831bf02da45a493fabf6e09d1bd8f`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:07.272640 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-9248b7df-49cf-486c-9b26-c8c46fff285c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:07.348083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1021/2023] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:07 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:07.352944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4403e86-708b-4429-964e-5b63a6b99bf8 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:07.427195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4403e86-708b-4429-964e-5b63a6b99bf8 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:07.449337 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c926721e-c3aa-4e33-b128-2322540f0c5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:07.449978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c926721e-c3aa-4e33-b128-2322540f0c5a admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:07.450296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c926721e-c3aa-4e33-b128-2322540f0c5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:07.450548 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c926721e-c3aa-4e33-b128-2322540f0c5a admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:07.451197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c926721e-c3aa-4e33-b128-2322540f0c5a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:07.453261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c926721e-c3aa-4e33-b128-2322540f0c5a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:07.513226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1003/2024] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:07 2022] POST /identity/v3/projects => generated 375 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:07.514766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9248b7df-49cf-486c-9b26-c8c46fff285c None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:07.532015 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9248b7df-49cf-486c-9b26-c8c46fff285c None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:07.613019 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79a4f84-6cf6-4f28-93e7-821e09d5dc3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:07.613826 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79a4f84-6cf6-4f28-93e7-821e09d5dc3d admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:07.614274 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79a4f84-6cf6-4f28-93e7-821e09d5dc3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:07.614697 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79a4f84-6cf6-4f28-93e7-821e09d5dc3d admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:07.615406 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79a4f84-6cf6-4f28-93e7-821e09d5dc3d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:07.617487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79a4f84-6cf6-4f28-93e7-821e09d5dc3d admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:07.648668 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1022/2025] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:07 2022] POST /identity/v3/users => generated 294 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:07.652712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c926721e-c3aa-4e33-b128-2322540f0c5a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:07.664143 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c926721e-c3aa-4e33-b128-2322540f0c5a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:07.671282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355c9ef9-aca2-4613-bac6-5c55dc625063 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:07.671999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355c9ef9-aca2-4613-bac6-5c55dc625063 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:07.671999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355c9ef9-aca2-4613-bac6-5c55dc625063 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:07.672276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355c9ef9-aca2-4613-bac6-5c55dc625063 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:07.672956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355c9ef9-aca2-4613-bac6-5c55dc625063 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:07.677258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355c9ef9-aca2-4613-bac6-5c55dc625063 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:07.717358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1004/2026] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:07 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:07.721925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79a4f84-6cf6-4f28-93e7-821e09d5dc3d admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:07.729885 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:07.738829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79a4f84-6cf6-4f28-93e7-821e09d5dc3d admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:07.750733 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd78b4c8-6da7-43ae-af11-5678363caeca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:07.751419 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd78b4c8-6da7-43ae-af11-5678363caeca admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:07.751701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd78b4c8-6da7-43ae-af11-5678363caeca admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:07.752045 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd78b4c8-6da7-43ae-af11-5678363caeca admin admin] PATH_INFO: `/v3/projects/51bcf68212a4419e9378e2c8a2bee261/users/c67997ff0ed54e1d8e931de06018a225/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:07.797754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd78b4c8-6da7-43ae-af11-5678363caeca admin admin] RBAC: Authorizing `identity:create_grant(project_id=51bcf68212a4419e9378e2c8a2bee261, user_id=c67997ff0ed54e1d8e931de06018a225, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:07.800999 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd78b4c8-6da7-43ae-af11-5678363caeca admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:07.834787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1023/2027] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:07 2022] PUT /identity/v3/projects/51bcf68212a4419e9378e2c8a2bee261/users/c67997ff0ed54e1d8e931de06018a225/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:07.836663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355c9ef9-aca2-4613-bac6-5c55dc625063 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:07.856768 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355c9ef9-aca2-4613-bac6-5c55dc625063 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:07.954709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75979c5f-e4d9-4998-bf50-0696453679f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:07.955368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75979c5f-e4d9-4998-bf50-0696453679f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:07.955647 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75979c5f-e4d9-4998-bf50-0696453679f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:07.955921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75979c5f-e4d9-4998-bf50-0696453679f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:07.956577 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75979c5f-e4d9-4998-bf50-0696453679f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:07.958218 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75979c5f-e4d9-4998-bf50-0696453679f5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:07.970385 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1005/2028] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:07 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:07.974777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd78b4c8-6da7-43ae-af11-5678363caeca admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:08.020613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd78b4c8-6da7-43ae-af11-5678363caeca admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:08.027052 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743253a2-f0e0-443f-91f1-8f06a6dd79d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:08.027741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743253a2-f0e0-443f-91f1-8f06a6dd79d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:08.029372 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743253a2-f0e0-443f-91f1-8f06a6dd79d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:08.030194 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743253a2-f0e0-443f-91f1-8f06a6dd79d9 admin admin] PATH_INFO: `/v3/projects/51bcf68212a4419e9378e2c8a2bee261/users/c67997ff0ed54e1d8e931de06018a225/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:08.034198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-743253a2-f0e0-443f-91f1-8f06a6dd79d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=51bcf68212a4419e9378e2c8a2bee261, user_id=c67997ff0ed54e1d8e931de06018a225, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:08.038813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-743253a2-f0e0-443f-91f1-8f06a6dd79d9 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:08.061645 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1024/2029] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:07 2022] PUT /identity/v3/projects/51bcf68212a4419e9378e2c8a2bee261/users/c67997ff0ed54e1d8e931de06018a225/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:08.073601 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5d2f9d-7b36-42e4-926a-e3b0b197529b None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:08.073945 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5d2f9d-7b36-42e4-926a-e3b0b197529b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:08.074577 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5d2f9d-7b36-42e4-926a-e3b0b197529b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:08.116313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-fa5d2f9d-7b36-42e4-926a-e3b0b197529b None None] MFA Rules not processed for user `c67997ff0ed54e1d8e931de06018a225`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:08.118744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fa5d2f9d-7b36-42e4-926a-e3b0b197529b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:08.181896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:08.309106 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1006/2030] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:08 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:08.410907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743253a2-f0e0-443f-91f1-8f06a6dd79d9 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:08.478658 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743253a2-f0e0-443f-91f1-8f06a6dd79d9 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:08.483343 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895116df-49ad-4b86-840c-af75ad7915d0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:08.483954 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895116df-49ad-4b86-840c-af75ad7915d0 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:08.484803 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895116df-49ad-4b86-840c-af75ad7915d0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:08.484803 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895116df-49ad-4b86-840c-af75ad7915d0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:08.565071 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895116df-49ad-4b86-840c-af75ad7915d0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:08.567405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895116df-49ad-4b86-840c-af75ad7915d0 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:08.581668 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1025/2031] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:08 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:09.870482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5d2f9d-7b36-42e4-926a-e3b0b197529b None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:09.871981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fa5d2f9d-7b36-42e4-926a-e3b0b197529b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:09.951974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5d2f9d-7b36-42e4-926a-e3b0b197529b None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:09.959029 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7baeb51a-ef10-42c9-bd7f-093b408c657f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:09.959441 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baeb51a-ef10-42c9-bd7f-093b408c657f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:09.960094 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baeb51a-ef10-42c9-bd7f-093b408c657f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:09.960094 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baeb51a-ef10-42c9-bd7f-093b408c657f admin admin] PATH_INFO: `/v3/users/cb01ad6db79947a79a14f1884ccb98cd` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:09.961148 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7baeb51a-ef10-42c9-bd7f-093b408c657f admin admin] RBAC: Authorizing `identity:delete_user(user_id=cb01ad6db79947a79a14f1884ccb98cd)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:09.963093 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7baeb51a-ef10-42c9-bd7f-093b408c657f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:10.062433 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7baeb51a-ef10-42c9-bd7f-093b408c657f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cb01ad6db79947a79a14f1884ccb98cd'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:10.077750 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7baeb51a-ef10-42c9-bd7f-093b408c657f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cb01ad6db79947a79a14f1884ccb98cd'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:10.087473 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7baeb51a-ef10-42c9-bd7f-093b408c657f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cb01ad6db79947a79a14f1884ccb98cd'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:10.103225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7baeb51a-ef10-42c9-bd7f-093b408c657f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cb01ad6db79947a79a14f1884ccb98cd'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:10.105665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7baeb51a-ef10-42c9-bd7f-093b408c657f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cb01ad6db79947a79a14f1884ccb98cd'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:10.120227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1007/2032] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:09 2022] DELETE /identity/v3/users/cb01ad6db79947a79a14f1884ccb98cd => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:10.130574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e538ffc-f207-415b-96cf-014e953f93a2 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:10.130987 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e538ffc-f207-415b-96cf-014e953f93a2 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:10.131174 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e538ffc-f207-415b-96cf-014e953f93a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:10.159301 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-4e538ffc-f207-415b-96cf-014e953f93a2 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:10.161000 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4e538ffc-f207-415b-96cf-014e953f93a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:10.251012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1026/2033] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:10.258355 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7baeb51a-ef10-42c9-bd7f-093b408c657f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:10.259999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7baeb51a-ef10-42c9-bd7f-093b408c657f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:10.336168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7baeb51a-ef10-42c9-bd7f-093b408c657f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:10.341288 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb9f69e-19f0-45f0-ac0b-8f27315f904f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:10.341974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb9f69e-19f0-45f0-ac0b-8f27315f904f service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:10.342410 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb9f69e-19f0-45f0-ac0b-8f27315f904f service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:10.342764 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb9f69e-19f0-45f0-ac0b-8f27315f904f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:10.363745 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb9f69e-19f0-45f0-ac0b-8f27315f904f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:10.363745 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb9f69e-19f0-45f0-ac0b-8f27315f904f service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:10.366646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:10.378922 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1008/2034] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:10.875151 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e538ffc-f207-415b-96cf-014e953f93a2 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:10.876803 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4e538ffc-f207-415b-96cf-014e953f93a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:10.889659 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e538ffc-f207-415b-96cf-014e953f93a2 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:10.895846 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40fc538-8515-4647-abc0-4b29dbf4cae9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:10.896578 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40fc538-8515-4647-abc0-4b29dbf4cae9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:10.896956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40fc538-8515-4647-abc0-4b29dbf4cae9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:10.897377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40fc538-8515-4647-abc0-4b29dbf4cae9 admin admin] PATH_INFO: `/v3/projects/b5e2d9838e6f4161a4f6a05d6cab3f3a` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:10.898677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40fc538-8515-4647-abc0-4b29dbf4cae9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b5e2d9838e6f4161a4f6a05d6cab3f3a)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:10.900712 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40fc538-8515-4647-abc0-4b29dbf4cae9 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:10.919441 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:10.959365 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c40fc538-8515-4647-abc0-4b29dbf4cae9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b5e2d9838e6f4161a4f6a05d6cab3f3a', 'request_id': 'req-c40fc538-8515-4647-abc0-4b29dbf4cae9'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:10.966634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c40fc538-8515-4647-abc0-4b29dbf4cae9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b5e2d9838e6f4161a4f6a05d6cab3f3a', 'request_id': 'req-c40fc538-8515-4647-abc0-4b29dbf4cae9'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:10.975696 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c40fc538-8515-4647-abc0-4b29dbf4cae9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b5e2d9838e6f4161a4f6a05d6cab3f3a', 'request_id': 'req-c40fc538-8515-4647-abc0-4b29dbf4cae9'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:10.988060 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c40fc538-8515-4647-abc0-4b29dbf4cae9 admin admin] The token cache is being invalidate because project b5e2d9838e6f4161a4f6a05d6cab3f3a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:10.988350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c40fc538-8515-4647-abc0-4b29dbf4cae9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:10.989004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c40fc538-8515-4647-abc0-4b29dbf4cae9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:10.990534 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1027/2035] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:10 2022] DELETE /identity/v3/projects/b5e2d9838e6f4161a4f6a05d6cab3f3a => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:10.994949 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb9f69e-19f0-45f0-ac0b-8f27315f904f service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:11.006350 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb9f69e-19f0-45f0-ac0b-8f27315f904f service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:11.065156 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383746d8-db9e-4e74-87df-5bd6d9baf9aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:11.065904 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383746d8-db9e-4e74-87df-5bd6d9baf9aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:11.066258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383746d8-db9e-4e74-87df-5bd6d9baf9aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:11.066640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383746d8-db9e-4e74-87df-5bd6d9baf9aa admin admin] PATH_INFO: `/v3/users/b332ab4eac3e46a791671861ab9bec80` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:11.067803 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383746d8-db9e-4e74-87df-5bd6d9baf9aa admin admin] RBAC: Authorizing `identity:delete_user(user_id=b332ab4eac3e46a791671861ab9bec80)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:11.069850 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383746d8-db9e-4e74-87df-5bd6d9baf9aa admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:11.131807 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-383746d8-db9e-4e74-87df-5bd6d9baf9aa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b332ab4eac3e46a791671861ab9bec80'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.144686 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-383746d8-db9e-4e74-87df-5bd6d9baf9aa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b332ab4eac3e46a791671861ab9bec80'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.151997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-383746d8-db9e-4e74-87df-5bd6d9baf9aa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b332ab4eac3e46a791671861ab9bec80'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.159252 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-383746d8-db9e-4e74-87df-5bd6d9baf9aa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b332ab4eac3e46a791671861ab9bec80'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.159862 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-383746d8-db9e-4e74-87df-5bd6d9baf9aa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b332ab4eac3e46a791671861ab9bec80'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.165953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1009/2036] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:10 2022] DELETE /identity/v3/users/b332ab4eac3e46a791671861ab9bec80 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:11.315260 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40fc538-8515-4647-abc0-4b29dbf4cae9 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:11.316938 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c40fc538-8515-4647-abc0-4b29dbf4cae9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:11.332302 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40fc538-8515-4647-abc0-4b29dbf4cae9 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:11.392570 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f3cea7-ec26-4693-be91-8ba8cae32fb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:11.393249 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f3cea7-ec26-4693-be91-8ba8cae32fb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:11.393535 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f3cea7-ec26-4693-be91-8ba8cae32fb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:11.393813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f3cea7-ec26-4693-be91-8ba8cae32fb2 admin admin] PATH_INFO: `/v3/users/ffa831bf02da45a493fabf6e09d1bd8f` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:11.394954 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f3cea7-ec26-4693-be91-8ba8cae32fb2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ffa831bf02da45a493fabf6e09d1bd8f)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:11.396932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f3cea7-ec26-4693-be91-8ba8cae32fb2 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:11.447037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383746d8-db9e-4e74-87df-5bd6d9baf9aa admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:11.448556 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-383746d8-db9e-4e74-87df-5bd6d9baf9aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:11.456216 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a9f3cea7-ec26-4693-be91-8ba8cae32fb2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ffa831bf02da45a493fabf6e09d1bd8f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.464374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383746d8-db9e-4e74-87df-5bd6d9baf9aa admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:11.466950 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a9f3cea7-ec26-4693-be91-8ba8cae32fb2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ffa831bf02da45a493fabf6e09d1bd8f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.468121 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a9f3cea7-ec26-4693-be91-8ba8cae32fb2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ffa831bf02da45a493fabf6e09d1bd8f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.473112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd1ce9e-8214-4f1b-8cf8-b3c2fa320d54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:11.473993 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd1ce9e-8214-4f1b-8cf8-b3c2fa320d54 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:11.477437 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd1ce9e-8214-4f1b-8cf8-b3c2fa320d54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:11.478054 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd1ce9e-8214-4f1b-8cf8-b3c2fa320d54 admin admin] PATH_INFO: `/v3/projects/03fb623eee534c9bb86ec43fde16eed9` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:11.479336 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd1ce9e-8214-4f1b-8cf8-b3c2fa320d54 admin admin] RBAC: Authorizing `identity:delete_project(project_id=03fb623eee534c9bb86ec43fde16eed9)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:11.481503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd1ce9e-8214-4f1b-8cf8-b3c2fa320d54 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:11.485498 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a9f3cea7-ec26-4693-be91-8ba8cae32fb2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ffa831bf02da45a493fabf6e09d1bd8f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.486713 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a9f3cea7-ec26-4693-be91-8ba8cae32fb2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ffa831bf02da45a493fabf6e09d1bd8f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.509201 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1028/2037] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:11 2022] DELETE /identity/v3/users/ffa831bf02da45a493fabf6e09d1bd8f => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:11.515006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5622305-1b49-493b-b65c-ee123fd0dfaf None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:11.515667 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5622305-1b49-493b-b65c-ee123fd0dfaf None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:11.516114 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5622305-1b49-493b-b65c-ee123fd0dfaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:11.544203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-c5622305-1b49-493b-b65c-ee123fd0dfaf None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:11.546057 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c5622305-1b49-493b-b65c-ee123fd0dfaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:11.566009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2bd1ce9e-8214-4f1b-8cf8-b3c2fa320d54 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '03fb623eee534c9bb86ec43fde16eed9', 'request_id': 'req-2bd1ce9e-8214-4f1b-8cf8-b3c2fa320d54'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.576753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:11.577517 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2bd1ce9e-8214-4f1b-8cf8-b3c2fa320d54 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '03fb623eee534c9bb86ec43fde16eed9', 'request_id': 'req-2bd1ce9e-8214-4f1b-8cf8-b3c2fa320d54'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.587795 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2bd1ce9e-8214-4f1b-8cf8-b3c2fa320d54 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '03fb623eee534c9bb86ec43fde16eed9', 'request_id': 'req-2bd1ce9e-8214-4f1b-8cf8-b3c2fa320d54'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.597713 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:11.602905 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2bd1ce9e-8214-4f1b-8cf8-b3c2fa320d54 admin admin] The token cache is being invalidate because project 03fb623eee534c9bb86ec43fde16eed9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:11.603202 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2bd1ce9e-8214-4f1b-8cf8-b3c2fa320d54 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.604221 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2bd1ce9e-8214-4f1b-8cf8-b3c2fa320d54 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.606804 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1010/2038] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:11 2022] DELETE /identity/v3/projects/03fb623eee534c9bb86ec43fde16eed9 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:11.611050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd1ce9e-8214-4f1b-8cf8-b3c2fa320d54 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:11.614047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2bd1ce9e-8214-4f1b-8cf8-b3c2fa320d54 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:11.632698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1029/2039] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:11.636732 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5622305-1b49-493b-b65c-ee123fd0dfaf None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:11.638633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c5622305-1b49-493b-b65c-ee123fd0dfaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:11.653395 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5622305-1b49-493b-b65c-ee123fd0dfaf None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:11.662828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddace0f6-df78-4fb4-9f2a-fcdeea381377 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:11.663469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddace0f6-df78-4fb4-9f2a-fcdeea381377 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:11.663747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddace0f6-df78-4fb4-9f2a-fcdeea381377 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:11.664017 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddace0f6-df78-4fb4-9f2a-fcdeea381377 admin admin] PATH_INFO: `/v3/users/e395aff2c86d46eca9d061a1658145df` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:11.665526 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddace0f6-df78-4fb4-9f2a-fcdeea381377 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e395aff2c86d46eca9d061a1658145df)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:11.667455 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddace0f6-df78-4fb4-9f2a-fcdeea381377 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:11.702725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd1ce9e-8214-4f1b-8cf8-b3c2fa320d54 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:11.712926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39dbf937-a82f-493e-bf7d-9936ac944a1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:11.713633 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dbf937-a82f-493e-bf7d-9936ac944a1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:11.713930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dbf937-a82f-493e-bf7d-9936ac944a1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:11.714202 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dbf937-a82f-493e-bf7d-9936ac944a1f admin admin] PATH_INFO: `/v3/users/9dfedf5a775944b1b8b665e5654efe8b` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:11.715480 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39dbf937-a82f-493e-bf7d-9936ac944a1f admin admin] RBAC: Authorizing `identity:delete_user(user_id=9dfedf5a775944b1b8b665e5654efe8b)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:11.717526 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39dbf937-a82f-493e-bf7d-9936ac944a1f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:11.728277 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ddace0f6-df78-4fb4-9f2a-fcdeea381377 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e395aff2c86d46eca9d061a1658145df'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.742940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ddace0f6-df78-4fb4-9f2a-fcdeea381377 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e395aff2c86d46eca9d061a1658145df'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.743845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ddace0f6-df78-4fb4-9f2a-fcdeea381377 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e395aff2c86d46eca9d061a1658145df'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.761429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ddace0f6-df78-4fb4-9f2a-fcdeea381377 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e395aff2c86d46eca9d061a1658145df'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.762374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ddace0f6-df78-4fb4-9f2a-fcdeea381377 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e395aff2c86d46eca9d061a1658145df'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.777514 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1030/2040] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:11 2022] DELETE /identity/v3/users/e395aff2c86d46eca9d061a1658145df => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:11.778295 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddace0f6-df78-4fb4-9f2a-fcdeea381377 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:11.782550 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ddace0f6-df78-4fb4-9f2a-fcdeea381377 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:11.789065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-39dbf937-a82f-493e-bf7d-9936ac944a1f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9dfedf5a775944b1b8b665e5654efe8b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.805802 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-39dbf937-a82f-493e-bf7d-9936ac944a1f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9dfedf5a775944b1b8b665e5654efe8b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.838283 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-39dbf937-a82f-493e-bf7d-9936ac944a1f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9dfedf5a775944b1b8b665e5654efe8b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.852803 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-39dbf937-a82f-493e-bf7d-9936ac944a1f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9dfedf5a775944b1b8b665e5654efe8b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.853595 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-39dbf937-a82f-493e-bf7d-9936ac944a1f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9dfedf5a775944b1b8b665e5654efe8b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:11.865687 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1011/2041] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:11 2022] DELETE /identity/v3/users/9dfedf5a775944b1b8b665e5654efe8b => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:11.870957 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f2ecd7-0f69-4a2d-9da7-acbe46969fe0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:11.871302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f2ecd7-0f69-4a2d-9da7-acbe46969fe0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:11.871849 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f2ecd7-0f69-4a2d-9da7-acbe46969fe0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:11.909811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddace0f6-df78-4fb4-9f2a-fcdeea381377 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:11.912713 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-22f2ecd7-0f69-4a2d-9da7-acbe46969fe0 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:11.914073 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-22f2ecd7-0f69-4a2d-9da7-acbe46969fe0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:11.996248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37aa139-0244-4afa-bd38-46cc17a1411f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:11.997373 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37aa139-0244-4afa-bd38-46cc17a1411f service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:11.997373 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37aa139-0244-4afa-bd38-46cc17a1411f service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:11.997596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37aa139-0244-4afa-bd38-46cc17a1411f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:11.999971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e37aa139-0244-4afa-bd38-46cc17a1411f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:12.009251 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:12.019041 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1012/2042] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:27:12.040063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f2ecd7-0f69-4a2d-9da7-acbe46969fe0 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:12.041571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-22f2ecd7-0f69-4a2d-9da7-acbe46969fe0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:12.059758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f2ecd7-0f69-4a2d-9da7-acbe46969fe0 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:12.063912 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2eef4b7-f00f-4124-be41-2af58264e522 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:12.064698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2eef4b7-f00f-4124-be41-2af58264e522 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:12.064971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2eef4b7-f00f-4124-be41-2af58264e522 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:12.065267 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2eef4b7-f00f-4124-be41-2af58264e522 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:12.075394 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2eef4b7-f00f-4124-be41-2af58264e522 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:12.076854 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2eef4b7-f00f-4124-be41-2af58264e522 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:12.094284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1013/2043] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:27:12.102209 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37aa139-0244-4afa-bd38-46cc17a1411f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:12.104875 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37aa139-0244-4afa-bd38-46cc17a1411f service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:12.121225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1031/2044] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:12.659466 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2eef4b7-f00f-4124-be41-2af58264e522 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:12.660895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f2eef4b7-f00f-4124-be41-2af58264e522 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:12.678535 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2eef4b7-f00f-4124-be41-2af58264e522 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:12.683894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339caf87-bfa1-4aa6-ba09-f72a4eb024e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:12.684656 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339caf87-bfa1-4aa6-ba09-f72a4eb024e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:12.685030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339caf87-bfa1-4aa6-ba09-f72a4eb024e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:12.685434 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339caf87-bfa1-4aa6-ba09-f72a4eb024e5 admin admin] PATH_INFO: `/v3/projects/1e76228c6087450bae92877889c8d58c` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:12.686689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339caf87-bfa1-4aa6-ba09-f72a4eb024e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1e76228c6087450bae92877889c8d58c)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:12.689854 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339caf87-bfa1-4aa6-ba09-f72a4eb024e5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:12.691675 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:12.758250 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-339caf87-bfa1-4aa6-ba09-f72a4eb024e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1e76228c6087450bae92877889c8d58c', 'request_id': 'req-339caf87-bfa1-4aa6-ba09-f72a4eb024e5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:12.764545 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-339caf87-bfa1-4aa6-ba09-f72a4eb024e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1e76228c6087450bae92877889c8d58c', 'request_id': 'req-339caf87-bfa1-4aa6-ba09-f72a4eb024e5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:12.776911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-339caf87-bfa1-4aa6-ba09-f72a4eb024e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1e76228c6087450bae92877889c8d58c', 'request_id': 'req-339caf87-bfa1-4aa6-ba09-f72a4eb024e5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:12.791705 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-339caf87-bfa1-4aa6-ba09-f72a4eb024e5 admin admin] The token cache is being invalidate because project 1e76228c6087450bae92877889c8d58c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:12.792173 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-339caf87-bfa1-4aa6-ba09-f72a4eb024e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:12.792984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-339caf87-bfa1-4aa6-ba09-f72a4eb024e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:12.800598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1014/2045] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:12 2022] DELETE /identity/v3/projects/1e76228c6087450bae92877889c8d58c => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:12.803007 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc27539-5ce2-44df-99eb-67a3390c0741 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:12.803424 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc27539-5ce2-44df-99eb-67a3390c0741 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:12.803580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc27539-5ce2-44df-99eb-67a3390c0741 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:12.824889 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-4bc27539-5ce2-44df-99eb-67a3390c0741 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:12.826070 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4bc27539-5ce2-44df-99eb-67a3390c0741 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:12.916617 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1032/2046] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:27:12.919232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e48f08-edd3-49d0-89fe-90fd58c9d8a8 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:12.919531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e48f08-edd3-49d0-89fe-90fd58c9d8a8 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:12.919837 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e48f08-edd3-49d0-89fe-90fd58c9d8a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:12.982689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-21e48f08-edd3-49d0-89fe-90fd58c9d8a8 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:12.984059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-21e48f08-edd3-49d0-89fe-90fd58c9d8a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:12.997145 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1015/2047] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:12.998454 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc27539-5ce2-44df-99eb-67a3390c0741 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:13.009663 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc27539-5ce2-44df-99eb-67a3390c0741 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:13.016644 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e998a4-f835-463d-a926-e29cc8c41af3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:13.018011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e998a4-f835-463d-a926-e29cc8c41af3 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:13.018470 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e998a4-f835-463d-a926-e29cc8c41af3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:13.018912 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e998a4-f835-463d-a926-e29cc8c41af3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:13.019820 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e998a4-f835-463d-a926-e29cc8c41af3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:27:13.020186 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e998a4-f835-463d-a926-e29cc8c41af3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:13.021985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e998a4-f835-463d-a926-e29cc8c41af3 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:13.041083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1033/2048] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:27:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:27:13.045013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e48f08-edd3-49d0-89fe-90fd58c9d8a8 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:13.057314 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e48f08-edd3-49d0-89fe-90fd58c9d8a8 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:13.065142 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e998a4-f835-463d-a926-e29cc8c41af3 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:13.066729 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09a3b40-21b0-4c64-ba54-cea85bd0e544 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:13.067366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09a3b40-21b0-4c64-ba54-cea85bd0e544 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:13.067697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09a3b40-21b0-4c64-ba54-cea85bd0e544 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:13.068003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09a3b40-21b0-4c64-ba54-cea85bd0e544 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:13.068743 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09a3b40-21b0-4c64-ba54-cea85bd0e544 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:13.072019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09a3b40-21b0-4c64-ba54-cea85bd0e544 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:13.088832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-52e998a4-f835-463d-a926-e29cc8c41af3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:13.108432 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e998a4-f835-463d-a926-e29cc8c41af3 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:13.113920 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7441f0-e9de-4e17-84b7-86eb91bc5ccf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:13.114624 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7441f0-e9de-4e17-84b7-86eb91bc5ccf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:13.115345 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7441f0-e9de-4e17-84b7-86eb91bc5ccf admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:13.115345 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7441f0-e9de-4e17-84b7-86eb91bc5ccf admin admin] PATH_INFO: `/v3/projects/61d37e609d474ce4a6b628612a787172` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:13.116335 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7441f0-e9de-4e17-84b7-86eb91bc5ccf admin admin] RBAC: Authorizing `identity:delete_project(project_id=61d37e609d474ce4a6b628612a787172)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:13.119405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7441f0-e9de-4e17-84b7-86eb91bc5ccf admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:13.150117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1016/2049] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:13 2022] POST /identity/v3/projects => generated 361 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:13.151358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09a3b40-21b0-4c64-ba54-cea85bd0e544 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:13.154795 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d09a3b40-21b0-4c64-ba54-cea85bd0e544 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:13.195722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-9c7441f0-e9de-4e17-84b7-86eb91bc5ccf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '61d37e609d474ce4a6b628612a787172', 'request_id': 'req-9c7441f0-e9de-4e17-84b7-86eb91bc5ccf'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.202180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-9c7441f0-e9de-4e17-84b7-86eb91bc5ccf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '61d37e609d474ce4a6b628612a787172', 'request_id': 'req-9c7441f0-e9de-4e17-84b7-86eb91bc5ccf'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.213002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-9c7441f0-e9de-4e17-84b7-86eb91bc5ccf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '61d37e609d474ce4a6b628612a787172', 'request_id': 'req-9c7441f0-e9de-4e17-84b7-86eb91bc5ccf'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.234633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-9c7441f0-e9de-4e17-84b7-86eb91bc5ccf admin admin] The token cache is being invalidate because project 61d37e609d474ce4a6b628612a787172 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:13.234633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-9c7441f0-e9de-4e17-84b7-86eb91bc5ccf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.235516 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-9c7441f0-e9de-4e17-84b7-86eb91bc5ccf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.237338 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1034/2050] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:13 2022] DELETE /identity/v3/projects/61d37e609d474ce4a6b628612a787172 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:13.238174 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7441f0-e9de-4e17-84b7-86eb91bc5ccf admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:13.239861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-9c7441f0-e9de-4e17-84b7-86eb91bc5ccf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:13.260965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:13.265935 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09a3b40-21b0-4c64-ba54-cea85bd0e544 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:13.335674 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7441f0-e9de-4e17-84b7-86eb91bc5ccf admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:13.345357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce52e2cb-4c2d-426b-a555-67ba22cb47a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:13.346162 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce52e2cb-4c2d-426b-a555-67ba22cb47a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:13.346442 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce52e2cb-4c2d-426b-a555-67ba22cb47a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:13.346717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce52e2cb-4c2d-426b-a555-67ba22cb47a6 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:13.347352 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce52e2cb-4c2d-426b-a555-67ba22cb47a6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:13.350107 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce52e2cb-4c2d-426b-a555-67ba22cb47a6 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:13.361086 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f702b22-fdee-4885-b5c9-246e1160be7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:13.362470 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f702b22-fdee-4885-b5c9-246e1160be7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:13.362470 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f702b22-fdee-4885-b5c9-246e1160be7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:13.363001 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f702b22-fdee-4885-b5c9-246e1160be7f admin admin] PATH_INFO: `/v3/projects/2ef1fad6e4f7424abdb598e740902018` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:13.365113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f702b22-fdee-4885-b5c9-246e1160be7f admin admin] RBAC: Authorizing `identity:delete_project(project_id=2ef1fad6e4f7424abdb598e740902018)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:13.367624 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f702b22-fdee-4885-b5c9-246e1160be7f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:13.380604 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:13.383346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1035/2051] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:13 2022] POST /identity/v3/users => generated 281 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:13.385343 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fbb38e-2ad2-49cc-91ec-8cc9752790c5 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:13.388420 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fbb38e-2ad2-49cc-91ec-8cc9752790c5 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:13.388851 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fbb38e-2ad2-49cc-91ec-8cc9752790c5 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:13.390567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1036/2052] 10.222.0.85 () {54 vars in 921 bytes} [Tue Aug 30 04:27:13 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:27:13.392141 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fbb38e-2ad2-49cc-91ec-8cc9752790c5 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:13.395912 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f7fbb38e-2ad2-49cc-91ec-8cc9752790c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:13.417253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fbb38e-2ad2-49cc-91ec-8cc9752790c5 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:13.422776 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93200170-3fcf-4603-9918-af2c631ba441 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:13.423418 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93200170-3fcf-4603-9918-af2c631ba441 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:13.423931 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93200170-3fcf-4603-9918-af2c631ba441 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:13.424215 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93200170-3fcf-4603-9918-af2c631ba441 admin admin] PATH_INFO: `/v3/users/c67997ff0ed54e1d8e931de06018a225` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:13.425419 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93200170-3fcf-4603-9918-af2c631ba441 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c67997ff0ed54e1d8e931de06018a225)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:13.428251 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93200170-3fcf-4603-9918-af2c631ba441 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:13.452497 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4f702b22-fdee-4885-b5c9-246e1160be7f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2ef1fad6e4f7424abdb598e740902018', 'request_id': 'req-4f702b22-fdee-4885-b5c9-246e1160be7f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.461904 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4f702b22-fdee-4885-b5c9-246e1160be7f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2ef1fad6e4f7424abdb598e740902018', 'request_id': 'req-4f702b22-fdee-4885-b5c9-246e1160be7f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.472159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4f702b22-fdee-4885-b5c9-246e1160be7f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2ef1fad6e4f7424abdb598e740902018', 'request_id': 'req-4f702b22-fdee-4885-b5c9-246e1160be7f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.488191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4f702b22-fdee-4885-b5c9-246e1160be7f admin admin] The token cache is being invalidate because project 2ef1fad6e4f7424abdb598e740902018 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:13.488493 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4f702b22-fdee-4885-b5c9-246e1160be7f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.489214 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4f702b22-fdee-4885-b5c9-246e1160be7f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.490982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1017/2053] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:13 2022] DELETE /identity/v3/projects/2ef1fad6e4f7424abdb598e740902018 => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:13.491778 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f702b22-fdee-4885-b5c9-246e1160be7f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:13.493178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4f702b22-fdee-4885-b5c9-246e1160be7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:13.509782 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-93200170-3fcf-4603-9918-af2c631ba441 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c67997ff0ed54e1d8e931de06018a225'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.523693 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-93200170-3fcf-4603-9918-af2c631ba441 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c67997ff0ed54e1d8e931de06018a225'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.524515 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-93200170-3fcf-4603-9918-af2c631ba441 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c67997ff0ed54e1d8e931de06018a225'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.544794 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-93200170-3fcf-4603-9918-af2c631ba441 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c67997ff0ed54e1d8e931de06018a225'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.545906 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-93200170-3fcf-4603-9918-af2c631ba441 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c67997ff0ed54e1d8e931de06018a225'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.564127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1037/2054] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:13 2022] DELETE /identity/v3/users/c67997ff0ed54e1d8e931de06018a225 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:13.566345 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bd9ccb-bca5-4586-a059-b194aec4e69e None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:13.566653 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bd9ccb-bca5-4586-a059-b194aec4e69e None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:13.566927 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bd9ccb-bca5-4586-a059-b194aec4e69e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:13.578123 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f702b22-fdee-4885-b5c9-246e1160be7f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:13.595304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-84bd9ccb-bca5-4586-a059-b194aec4e69e None None] MFA Rules not processed for user `359d79cbfcad49beba2b495066824128`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:13.597062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-84bd9ccb-bca5-4586-a059-b194aec4e69e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:13.661512 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85d7931-a5cc-404f-a1a2-89f2da7cd94a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:13.662254 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85d7931-a5cc-404f-a1a2-89f2da7cd94a admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:13.662606 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85d7931-a5cc-404f-a1a2-89f2da7cd94a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:13.662960 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85d7931-a5cc-404f-a1a2-89f2da7cd94a admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:13.663695 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85d7931-a5cc-404f-a1a2-89f2da7cd94a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:13.666164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85d7931-a5cc-404f-a1a2-89f2da7cd94a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:13.682561 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1018/2055] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:13 2022] GET /identity/v3/roles => generated 1420 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:13.683765 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85d7931-a5cc-404f-a1a2-89f2da7cd94a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:13.686100 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d85d7931-a5cc-404f-a1a2-89f2da7cd94a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:13.701400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85d7931-a5cc-404f-a1a2-89f2da7cd94a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:13.711916 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f859ced-9c99-4b15-933c-98f105f4738e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:13.712980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f859ced-9c99-4b15-933c-98f105f4738e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:13.713529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f859ced-9c99-4b15-933c-98f105f4738e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:13.713956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f859ced-9c99-4b15-933c-98f105f4738e admin admin] PATH_INFO: `/v3/users/31f19605128c4df3b9d80d6e0ae7096d` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:13.714405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1038/2056] 10.222.0.85 () {58 vars in 1018 bytes} [Tue Aug 30 04:27:13 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:13.715360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bd9ccb-bca5-4586-a059-b194aec4e69e None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:13.716506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f859ced-9c99-4b15-933c-98f105f4738e admin admin] RBAC: Authorizing `identity:delete_user(user_id=31f19605128c4df3b9d80d6e0ae7096d)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:13.717908 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-84bd9ccb-bca5-4586-a059-b194aec4e69e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:13.720329 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f859ced-9c99-4b15-933c-98f105f4738e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:13.746996 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bd9ccb-bca5-4586-a059-b194aec4e69e None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:13.761818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22660903-752d-4b94-8116-ee58184dbede admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:13.762601 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22660903-752d-4b94-8116-ee58184dbede admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:13.762747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22660903-752d-4b94-8116-ee58184dbede admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:13.762992 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22660903-752d-4b94-8116-ee58184dbede admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:13.763683 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22660903-752d-4b94-8116-ee58184dbede admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:13.769717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22660903-752d-4b94-8116-ee58184dbede admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:13.774906 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:13.779228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:13.791515 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1039/2057] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:13 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:13.792619 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22660903-752d-4b94-8116-ee58184dbede admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:13.797328 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-22660903-752d-4b94-8116-ee58184dbede admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:13.809694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f859ced-9c99-4b15-933c-98f105f4738e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '31f19605128c4df3b9d80d6e0ae7096d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.820571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22660903-752d-4b94-8116-ee58184dbede admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:13.828670 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f859ced-9c99-4b15-933c-98f105f4738e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '31f19605128c4df3b9d80d6e0ae7096d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.829638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6993d4cd-a243-408d-a169-390bd4cff3f0 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '359d79cbfcad49beba2b495066824128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:13.830429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6993d4cd-a243-408d-a169-390bd4cff3f0 service placement] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:13.830736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6993d4cd-a243-408d-a169-390bd4cff3f0 service placement] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:13.831010 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6993d4cd-a243-408d-a169-390bd4cff3f0 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:13.837582 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f859ced-9c99-4b15-933c-98f105f4738e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '31f19605128c4df3b9d80d6e0ae7096d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.846935 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6993d4cd-a243-408d-a169-390bd4cff3f0 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:13.848739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6993d4cd-a243-408d-a169-390bd4cff3f0 service placement] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:13.859077 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f859ced-9c99-4b15-933c-98f105f4738e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '31f19605128c4df3b9d80d6e0ae7096d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.860148 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f859ced-9c99-4b15-933c-98f105f4738e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '31f19605128c4df3b9d80d6e0ae7096d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:13.864251 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1040/2058] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:13 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:13.873087 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1019/2059] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:13 2022] DELETE /identity/v3/users/31f19605128c4df3b9d80d6e0ae7096d => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:13.876475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6993d4cd-a243-408d-a169-390bd4cff3f0 service placement] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:13.897658 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6993d4cd-a243-408d-a169-390bd4cff3f0 service placement] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:13.992069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f859ced-9c99-4b15-933c-98f105f4738e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:13.995059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6f859ced-9c99-4b15-933c-98f105f4738e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:14.007836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566dcc93-509d-45ba-98f6-b9d768841528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:14.010449 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566dcc93-509d-45ba-98f6-b9d768841528 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:14.010859 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566dcc93-509d-45ba-98f6-b9d768841528 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:14.011147 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566dcc93-509d-45ba-98f6-b9d768841528 admin admin] PATH_INFO: `/v3/projects/91a5bb68aa1a4c4abe6ad691852e9dfe/users/05a855092bf94b6f9602c9e951a10079/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:14.038489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566dcc93-509d-45ba-98f6-b9d768841528 admin admin] RBAC: Authorizing `identity:create_grant(project_id=91a5bb68aa1a4c4abe6ad691852e9dfe, user_id=05a855092bf94b6f9602c9e951a10079, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:14.039340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f859ced-9c99-4b15-933c-98f105f4738e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:14.041267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566dcc93-509d-45ba-98f6-b9d768841528 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:14.059753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1041/2060] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:13 2022] PUT /identity/v3/projects/91a5bb68aa1a4c4abe6ad691852e9dfe/users/05a855092bf94b6f9602c9e951a10079/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:14.060085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084aa73a-5978-48b2-bd4d-3bc355be0053 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:14.061417 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084aa73a-5978-48b2-bd4d-3bc355be0053 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:14.063214 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084aa73a-5978-48b2-bd4d-3bc355be0053 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:14.064276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084aa73a-5978-48b2-bd4d-3bc355be0053 admin admin] PATH_INFO: `/v3/projects/51bcf68212a4419e9378e2c8a2bee261` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:14.072301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084aa73a-5978-48b2-bd4d-3bc355be0053 admin admin] RBAC: Authorizing `identity:delete_project(project_id=51bcf68212a4419e9378e2c8a2bee261)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:14.073613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dca9ef5-73f7-4625-8fb4-9178eafd37f8 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:14.073910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dca9ef5-73f7-4625-8fb4-9178eafd37f8 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:14.074221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dca9ef5-73f7-4625-8fb4-9178eafd37f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:14.074325 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084aa73a-5978-48b2-bd4d-3bc355be0053 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:14.138039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-2dca9ef5-73f7-4625-8fb4-9178eafd37f8 None None] MFA Rules not processed for user `05a855092bf94b6f9602c9e951a10079`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:14.139453 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-2dca9ef5-73f7-4625-8fb4-9178eafd37f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:14.159312 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-084aa73a-5978-48b2-bd4d-3bc355be0053 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '51bcf68212a4419e9378e2c8a2bee261', 'request_id': 'req-084aa73a-5978-48b2-bd4d-3bc355be0053'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.166952 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-084aa73a-5978-48b2-bd4d-3bc355be0053 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '51bcf68212a4419e9378e2c8a2bee261', 'request_id': 'req-084aa73a-5978-48b2-bd4d-3bc355be0053'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.178091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-084aa73a-5978-48b2-bd4d-3bc355be0053 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '51bcf68212a4419e9378e2c8a2bee261', 'request_id': 'req-084aa73a-5978-48b2-bd4d-3bc355be0053'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.199964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-084aa73a-5978-48b2-bd4d-3bc355be0053 admin admin] The token cache is being invalidate because project 51bcf68212a4419e9378e2c8a2bee261 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:14.200492 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-084aa73a-5978-48b2-bd4d-3bc355be0053 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.201393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-084aa73a-5978-48b2-bd4d-3bc355be0053 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.205240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1020/2061] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:13 2022] DELETE /identity/v3/projects/51bcf68212a4419e9378e2c8a2bee261 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:14.211493 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d7234f-82f2-4ce0-9f42-45e9899ffb32 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:14.211789 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d7234f-82f2-4ce0-9f42-45e9899ffb32 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:14.212068 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d7234f-82f2-4ce0-9f42-45e9899ffb32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:14.248704 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-50d7234f-82f2-4ce0-9f42-45e9899ffb32 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:14.250503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-50d7234f-82f2-4ce0-9f42-45e9899ffb32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:14.255082 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1042/2062] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:14 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:14.256219 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dca9ef5-73f7-4625-8fb4-9178eafd37f8 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:14.258593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-2dca9ef5-73f7-4625-8fb4-9178eafd37f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:14.367828 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1021/2063] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:27:14.369515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d7234f-82f2-4ce0-9f42-45e9899ffb32 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:14.371048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-50d7234f-82f2-4ce0-9f42-45e9899ffb32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:14.388636 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dca9ef5-73f7-4625-8fb4-9178eafd37f8 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:14.406304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c66634-a638-4162-a7e1-da4f4a119589 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:14.406304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c66634-a638-4162-a7e1-da4f4a119589 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:14.406304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c66634-a638-4162-a7e1-da4f4a119589 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:14.406304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c66634-a638-4162-a7e1-da4f4a119589 admin admin] PATH_INFO: `/v3/users/f6ff88ebcb8440e99539b0a2cda8e152` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:14.411632 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c66634-a638-4162-a7e1-da4f4a119589 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f6ff88ebcb8440e99539b0a2cda8e152)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:14.420455 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c66634-a638-4162-a7e1-da4f4a119589 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:14.426278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:14.451557 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:14.477476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d7234f-82f2-4ce0-9f42-45e9899ffb32 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:14.482304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabe91f3-3776-4dd2-b56c-66abfa9c6c1f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:14.482938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabe91f3-3776-4dd2-b56c-66abfa9c6c1f service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:14.483347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabe91f3-3776-4dd2-b56c-66abfa9c6c1f service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:14.483874 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabe91f3-3776-4dd2-b56c-66abfa9c6c1f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:14.497400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eabe91f3-3776-4dd2-b56c-66abfa9c6c1f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:14.499177 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eabe91f3-3776-4dd2-b56c-66abfa9c6c1f service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:14.499584 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f5c66634-a638-4162-a7e1-da4f4a119589 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f6ff88ebcb8440e99539b0a2cda8e152'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.508472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f5c66634-a638-4162-a7e1-da4f4a119589 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f6ff88ebcb8440e99539b0a2cda8e152'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.509305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f5c66634-a638-4162-a7e1-da4f4a119589 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f6ff88ebcb8440e99539b0a2cda8e152'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.510012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1022/2064] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:14 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:14.511448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0218f2e0-e85d-4506-b7a8-6944eeb2e8f0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:14.511733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0218f2e0-e85d-4506-b7a8-6944eeb2e8f0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:14.512069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0218f2e0-e85d-4506-b7a8-6944eeb2e8f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:14.526246 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f5c66634-a638-4162-a7e1-da4f4a119589 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f6ff88ebcb8440e99539b0a2cda8e152'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.527896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f5c66634-a638-4162-a7e1-da4f4a119589 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f6ff88ebcb8440e99539b0a2cda8e152'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.535121 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-0218f2e0-e85d-4506-b7a8-6944eeb2e8f0 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:14.536340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0218f2e0-e85d-4506-b7a8-6944eeb2e8f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:14.540263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1043/2065] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:14 2022] DELETE /identity/v3/users/f6ff88ebcb8440e99539b0a2cda8e152 => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:14.541148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c66634-a638-4162-a7e1-da4f4a119589 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:14.549207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f5c66634-a638-4162-a7e1-da4f4a119589 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:14.551443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1023/2066] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:14.552499 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0218f2e0-e85d-4506-b7a8-6944eeb2e8f0 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:14.554355 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0218f2e0-e85d-4506-b7a8-6944eeb2e8f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:14.572726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0218f2e0-e85d-4506-b7a8-6944eeb2e8f0 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:14.637375 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c66634-a638-4162-a7e1-da4f4a119589 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:14.643420 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87adcfa-33cd-48d9-b139-877a221c34af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:14.644669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87adcfa-33cd-48d9-b139-877a221c34af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:14.644941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87adcfa-33cd-48d9-b139-877a221c34af admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:14.645319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87adcfa-33cd-48d9-b139-877a221c34af admin admin] PATH_INFO: `/v3/projects/620391fe5ce24837a102c150548c9c82` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:14.646903 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87adcfa-33cd-48d9-b139-877a221c34af admin admin] RBAC: Authorizing `identity:delete_project(project_id=620391fe5ce24837a102c150548c9c82)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:14.648920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a45493a-e532-4565-bc5c-63a7e7e9eb57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:14.649607 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87adcfa-33cd-48d9-b139-877a221c34af admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:14.649718 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a45493a-e532-4565-bc5c-63a7e7e9eb57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:14.650549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a45493a-e532-4565-bc5c-63a7e7e9eb57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:14.650549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a45493a-e532-4565-bc5c-63a7e7e9eb57 admin admin] PATH_INFO: `/v3/users/05a855092bf94b6f9602c9e951a10079` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:14.651434 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a45493a-e532-4565-bc5c-63a7e7e9eb57 admin admin] RBAC: Authorizing `identity:delete_user(user_id=05a855092bf94b6f9602c9e951a10079)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:14.653416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a45493a-e532-4565-bc5c-63a7e7e9eb57 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:14.721888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5a45493a-e532-4565-bc5c-63a7e7e9eb57 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '05a855092bf94b6f9602c9e951a10079'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.727216 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b87adcfa-33cd-48d9-b139-877a221c34af admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '620391fe5ce24837a102c150548c9c82', 'request_id': 'req-b87adcfa-33cd-48d9-b139-877a221c34af'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.740680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b87adcfa-33cd-48d9-b139-877a221c34af admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '620391fe5ce24837a102c150548c9c82', 'request_id': 'req-b87adcfa-33cd-48d9-b139-877a221c34af'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.744713 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5a45493a-e532-4565-bc5c-63a7e7e9eb57 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '05a855092bf94b6f9602c9e951a10079'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.754886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5a45493a-e532-4565-bc5c-63a7e7e9eb57 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '05a855092bf94b6f9602c9e951a10079'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.766883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5a45493a-e532-4565-bc5c-63a7e7e9eb57 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '05a855092bf94b6f9602c9e951a10079'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.768360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5a45493a-e532-4565-bc5c-63a7e7e9eb57 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '05a855092bf94b6f9602c9e951a10079'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.771239 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b87adcfa-33cd-48d9-b139-877a221c34af admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '620391fe5ce24837a102c150548c9c82', 'request_id': 'req-b87adcfa-33cd-48d9-b139-877a221c34af'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.785522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1024/2067] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:14 2022] DELETE /identity/v3/users/05a855092bf94b6f9602c9e951a10079 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:14.787741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a45493a-e532-4565-bc5c-63a7e7e9eb57 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:14.793643 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5a45493a-e532-4565-bc5c-63a7e7e9eb57 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:14.801536 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b87adcfa-33cd-48d9-b139-877a221c34af admin admin] The token cache is being invalidate because project 620391fe5ce24837a102c150548c9c82 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:14.801929 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b87adcfa-33cd-48d9-b139-877a221c34af admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.802625 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b87adcfa-33cd-48d9-b139-877a221c34af admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:14.804492 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1044/2068] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:14 2022] DELETE /identity/v3/projects/620391fe5ce24837a102c150548c9c82 => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:14.808784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2ca3f7-a79f-495b-b464-e6490e281b09 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:14.808784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2ca3f7-a79f-495b-b464-e6490e281b09 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:14.808784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2ca3f7-a79f-495b-b464-e6490e281b09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:14.858792 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-7a2ca3f7-a79f-495b-b464-e6490e281b09 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:14.860188 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-7a2ca3f7-a79f-495b-b464-e6490e281b09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:14.866042 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:14.885146 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:14.923622 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a45493a-e532-4565-bc5c-63a7e7e9eb57 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:14.927249 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f722e7-8839-4bfa-90b2-7f39f9bc5ff3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:14.927960 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f722e7-8839-4bfa-90b2-7f39f9bc5ff3 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:14.928320 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f722e7-8839-4bfa-90b2-7f39f9bc5ff3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:14.929265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f722e7-8839-4bfa-90b2-7f39f9bc5ff3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:14.930997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f722e7-8839-4bfa-90b2-7f39f9bc5ff3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:27:14.930997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f722e7-8839-4bfa-90b2-7f39f9bc5ff3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:14.932582 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f722e7-8839-4bfa-90b2-7f39f9bc5ff3 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:14.946262 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1025/2069] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:27:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:27:14.947615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867a0a46-7210-4bca-969b-e791be19b636 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:14.947979 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867a0a46-7210-4bca-969b-e791be19b636 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:14.948333 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867a0a46-7210-4bca-969b-e791be19b636 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:14.963162 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1045/2070] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:14.964044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2ca3f7-a79f-495b-b464-e6490e281b09 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:14.965543 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-7a2ca3f7-a79f-495b-b464-e6490e281b09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:14.979615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-867a0a46-7210-4bca-969b-e791be19b636 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:14.981206 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-867a0a46-7210-4bca-969b-e791be19b636 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:14.983612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2ca3f7-a79f-495b-b464-e6490e281b09 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:14.989158 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d533f028-247f-407f-a9d3-fd1ac27d462f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:14.989822 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d533f028-247f-407f-a9d3-fd1ac27d462f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:14.990098 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d533f028-247f-407f-a9d3-fd1ac27d462f admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:14.990371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d533f028-247f-407f-a9d3-fd1ac27d462f admin admin] PATH_INFO: `/v3/projects/aa4e296e79654771af4e3b6bb19d402b` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:14.991476 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d533f028-247f-407f-a9d3-fd1ac27d462f admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa4e296e79654771af4e3b6bb19d402b)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:14.992859 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1026/2071] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:14.993595 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d533f028-247f-407f-a9d3-fd1ac27d462f admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:14.994258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867a0a46-7210-4bca-969b-e791be19b636 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:14.995701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-867a0a46-7210-4bca-969b-e791be19b636 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:15.008625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867a0a46-7210-4bca-969b-e791be19b636 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:15.013390 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8a0d7d-e858-46d3-a8d0-6605c7d22f81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:15.014035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8a0d7d-e858-46d3-a8d0-6605c7d22f81 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:15.014321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8a0d7d-e858-46d3-a8d0-6605c7d22f81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:15.014596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8a0d7d-e858-46d3-a8d0-6605c7d22f81 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:15.015236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8a0d7d-e858-46d3-a8d0-6605c7d22f81 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:15.017602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8a0d7d-e858-46d3-a8d0-6605c7d22f81 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:15.046139 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d533f028-247f-407f-a9d3-fd1ac27d462f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa4e296e79654771af4e3b6bb19d402b', 'request_id': 'req-d533f028-247f-407f-a9d3-fd1ac27d462f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:15.050857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1027/2072] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:14 2022] POST /identity/v3/projects => generated 383 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:15.050857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8a0d7d-e858-46d3-a8d0-6605c7d22f81 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:15.051104 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-de8a0d7d-e858-46d3-a8d0-6605c7d22f81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:15.053459 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d533f028-247f-407f-a9d3-fd1ac27d462f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa4e296e79654771af4e3b6bb19d402b', 'request_id': 'req-d533f028-247f-407f-a9d3-fd1ac27d462f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:15.064498 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d533f028-247f-407f-a9d3-fd1ac27d462f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa4e296e79654771af4e3b6bb19d402b', 'request_id': 'req-d533f028-247f-407f-a9d3-fd1ac27d462f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:15.076887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d533f028-247f-407f-a9d3-fd1ac27d462f admin admin] The token cache is being invalidate because project aa4e296e79654771af4e3b6bb19d402b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:15.077212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d533f028-247f-407f-a9d3-fd1ac27d462f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:15.077876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d533f028-247f-407f-a9d3-fd1ac27d462f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:15.079358 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1046/2073] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:14 2022] DELETE /identity/v3/projects/aa4e296e79654771af4e3b6bb19d402b => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:15.081147 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a49184-7254-4093-80b7-285a78fb608b None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:15.081429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a49184-7254-4093-80b7-285a78fb608b None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:15.081694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a49184-7254-4093-80b7-285a78fb608b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:15.109693 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-33a49184-7254-4093-80b7-285a78fb608b None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:15.111588 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-33a49184-7254-4093-80b7-285a78fb608b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:15.111767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8a0d7d-e858-46d3-a8d0-6605c7d22f81 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:15.127620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f36c9c-42ca-48be-9e52-b1e6b5404ad9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:15.127620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f36c9c-42ca-48be-9e52-b1e6b5404ad9 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:15.127620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f36c9c-42ca-48be-9e52-b1e6b5404ad9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:15.127620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f36c9c-42ca-48be-9e52-b1e6b5404ad9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:15.127928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d3f36c9c-42ca-48be-9e52-b1e6b5404ad9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:15.193216 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1047/2074] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:15.193814 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f36c9c-42ca-48be-9e52-b1e6b5404ad9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:15.194223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a49184-7254-4093-80b7-285a78fb608b None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:15.195284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f36c9c-42ca-48be-9e52-b1e6b5404ad9 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:15.198500 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-33a49184-7254-4093-80b7-285a78fb608b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:15.208591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1028/2075] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:15.209509 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f36c9c-42ca-48be-9e52-b1e6b5404ad9 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:15.211620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d3f36c9c-42ca-48be-9e52-b1e6b5404ad9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:15.215124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a49184-7254-4093-80b7-285a78fb608b None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:15.220391 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87858349-7c1b-4641-8168-1a6036bdc123 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:15.222015 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87858349-7c1b-4641-8168-1a6036bdc123 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:15.222015 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87858349-7c1b-4641-8168-1a6036bdc123 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:15.222240 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87858349-7c1b-4641-8168-1a6036bdc123 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:15.222742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87858349-7c1b-4641-8168-1a6036bdc123 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:15.224694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87858349-7c1b-4641-8168-1a6036bdc123 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:15.224845 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f36c9c-42ca-48be-9e52-b1e6b5404ad9 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:15.230243 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594e9f8e-1331-4c0c-a876-a565c70e0104 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:15.233128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594e9f8e-1331-4c0c-a876-a565c70e0104 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:15.233128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594e9f8e-1331-4c0c-a876-a565c70e0104 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:15.233128 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594e9f8e-1331-4c0c-a876-a565c70e0104 admin admin] PATH_INFO: `/v3/users/74618524351f46a7b41b7dbd3552cff3` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:15.234790 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594e9f8e-1331-4c0c-a876-a565c70e0104 admin admin] RBAC: Authorizing `identity:delete_user(user_id=74618524351f46a7b41b7dbd3552cff3)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:15.236883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594e9f8e-1331-4c0c-a876-a565c70e0104 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:15.252258 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1048/2076] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:15 2022] POST /identity/v3/users => generated 292 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:15.253236 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87858349-7c1b-4641-8168-1a6036bdc123 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:15.263084 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87858349-7c1b-4641-8168-1a6036bdc123 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:15.269529 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd3acca-b5be-45e5-8fa3-c8b09b78b74f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:15.270143 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd3acca-b5be-45e5-8fa3-c8b09b78b74f admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:15.271031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd3acca-b5be-45e5-8fa3-c8b09b78b74f admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:15.271361 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd3acca-b5be-45e5-8fa3-c8b09b78b74f admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:15.271985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd3acca-b5be-45e5-8fa3-c8b09b78b74f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:27:15.272442 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd3acca-b5be-45e5-8fa3-c8b09b78b74f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:15.274685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd3acca-b5be-45e5-8fa3-c8b09b78b74f admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:15.297130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1049/2077] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:27:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:27:15.298049 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd3acca-b5be-45e5-8fa3-c8b09b78b74f admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:15.299572 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-594e9f8e-1331-4c0c-a876-a565c70e0104 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '74618524351f46a7b41b7dbd3552cff3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:15.304394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:15.308646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd3acca-b5be-45e5-8fa3-c8b09b78b74f admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:15.317265 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c9f1ef-7e17-4736-a278-01fd86611bc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:15.317530 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:INSERT=1,keystone:DELETE=9 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:15.317961 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c9f1ef-7e17-4736-a278-01fd86611bc6 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:15.318656 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c9f1ef-7e17-4736-a278-01fd86611bc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:15.318939 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c9f1ef-7e17-4736-a278-01fd86611bc6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:15.319872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c9f1ef-7e17-4736-a278-01fd86611bc6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:15.322032 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c9f1ef-7e17-4736-a278-01fd86611bc6 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:15.322412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-594e9f8e-1331-4c0c-a876-a565c70e0104 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '74618524351f46a7b41b7dbd3552cff3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:15.335498 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-594e9f8e-1331-4c0c-a876-a565c70e0104 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '74618524351f46a7b41b7dbd3552cff3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:15.338318 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1050/2078] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:15 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:15.338971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c9f1ef-7e17-4736-a278-01fd86611bc6 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:15.348704 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-594e9f8e-1331-4c0c-a876-a565c70e0104 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '74618524351f46a7b41b7dbd3552cff3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:15.349587 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-594e9f8e-1331-4c0c-a876-a565c70e0104 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '74618524351f46a7b41b7dbd3552cff3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:15.356910 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1029/2079] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:15 2022] DELETE /identity/v3/users/74618524351f46a7b41b7dbd3552cff3 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:15.358292 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594e9f8e-1331-4c0c-a876-a565c70e0104 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:15.359280 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-594e9f8e-1331-4c0c-a876-a565c70e0104 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:15.361470 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c9f1ef-7e17-4736-a278-01fd86611bc6 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:15.449123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ba911b-43fa-4c2c-b910-ddee86ccccb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:15.450100 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ba911b-43fa-4c2c-b910-ddee86ccccb4 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:15.450552 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ba911b-43fa-4c2c-b910-ddee86ccccb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:15.450933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ba911b-43fa-4c2c-b910-ddee86ccccb4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:15.452034 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ba911b-43fa-4c2c-b910-ddee86ccccb4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:15.455237 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ba911b-43fa-4c2c-b910-ddee86ccccb4 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:15.468965 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594e9f8e-1331-4c0c-a876-a565c70e0104 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:15.476860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1036f123-0790-4fd4-8c19-007256b5d745 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:15.477773 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1036f123-0790-4fd4-8c19-007256b5d745 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:15.478219 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1036f123-0790-4fd4-8c19-007256b5d745 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:15.478651 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1036f123-0790-4fd4-8c19-007256b5d745 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:15.479530 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1036f123-0790-4fd4-8c19-007256b5d745 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:15.481879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1036f123-0790-4fd4-8c19-007256b5d745 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:15.501092 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1051/2080] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:15 2022] POST /identity/v3/projects => generated 393 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:15.504777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1030/2081] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:15 2022] GET /identity/v3/roles => generated 1420 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:15.505955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ba911b-43fa-4c2c-b910-ddee86ccccb4 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:15.508421 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-73ba911b-43fa-4c2c-b910-ddee86ccccb4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:15.513228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1036f123-0790-4fd4-8c19-007256b5d745 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:15.533521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1036f123-0790-4fd4-8c19-007256b5d745 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:15.603869 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ba911b-43fa-4c2c-b910-ddee86ccccb4 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:15.610489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54e104a-f284-4222-a304-21c1cb31448c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:15.611627 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54e104a-f284-4222-a304-21c1cb31448c admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:15.612011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54e104a-f284-4222-a304-21c1cb31448c admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:15.612351 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54e104a-f284-4222-a304-21c1cb31448c admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:15.613260 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54e104a-f284-4222-a304-21c1cb31448c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:15.615976 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54e104a-f284-4222-a304-21c1cb31448c admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:15.661905 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1052/2082] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:15 2022] POST /identity/v3/users => generated 297 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:15.667064 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea334cbf-42e0-4c4e-8b30-320316f619f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:15.668294 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea334cbf-42e0-4c4e-8b30-320316f619f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:15.668294 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea334cbf-42e0-4c4e-8b30-320316f619f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:15.668746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea334cbf-42e0-4c4e-8b30-320316f619f4 admin admin] PATH_INFO: `/v3/projects/2d8842fc1da94bf681eeb42464032b8a/users/3a25d172babb4cd698fac79d7412d25b/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:15.671328 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54e104a-f284-4222-a304-21c1cb31448c admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:15.674392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-a54e104a-f284-4222-a304-21c1cb31448c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:15.698369 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54e104a-f284-4222-a304-21c1cb31448c admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:15.704258 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf8e840-5c0d-4ce5-a964-60ba480a803a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:15.706017 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf8e840-5c0d-4ce5-a964-60ba480a803a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:15.706017 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf8e840-5c0d-4ce5-a964-60ba480a803a admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:15.706244 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf8e840-5c0d-4ce5-a964-60ba480a803a admin admin] PATH_INFO: `/v3/projects/83c45254f5f4416eb1bc3bbba7242e4b` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:15.708894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea334cbf-42e0-4c4e-8b30-320316f619f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d8842fc1da94bf681eeb42464032b8a, user_id=3a25d172babb4cd698fac79d7412d25b, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:15.711984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea334cbf-42e0-4c4e-8b30-320316f619f4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:15.717304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf8e840-5c0d-4ce5-a964-60ba480a803a admin admin] RBAC: Authorizing `identity:delete_project(project_id=83c45254f5f4416eb1bc3bbba7242e4b)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:15.722948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf8e840-5c0d-4ce5-a964-60ba480a803a admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:15.732143 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1031/2083] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:15 2022] PUT /identity/v3/projects/2d8842fc1da94bf681eeb42464032b8a/users/3a25d172babb4cd698fac79d7412d25b/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:15.732143 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea334cbf-42e0-4c4e-8b30-320316f619f4 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:15.823800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea334cbf-42e0-4c4e-8b30-320316f619f4 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:15.827347 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bdf8e840-5c0d-4ce5-a964-60ba480a803a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '83c45254f5f4416eb1bc3bbba7242e4b', 'request_id': 'req-bdf8e840-5c0d-4ce5-a964-60ba480a803a'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:15.827743 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=2,keystone:DELETE=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:15.838257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bdf8e840-5c0d-4ce5-a964-60ba480a803a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '83c45254f5f4416eb1bc3bbba7242e4b', 'request_id': 'req-bdf8e840-5c0d-4ce5-a964-60ba480a803a'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:15.841572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:INSERT=5,keystone:DELETE=6 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:15.851236 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bdf8e840-5c0d-4ce5-a964-60ba480a803a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '83c45254f5f4416eb1bc3bbba7242e4b', 'request_id': 'req-bdf8e840-5c0d-4ce5-a964-60ba480a803a'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:15.866137 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bdf8e840-5c0d-4ce5-a964-60ba480a803a admin admin] The token cache is being invalidate because project 83c45254f5f4416eb1bc3bbba7242e4b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:15.867309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bdf8e840-5c0d-4ce5-a964-60ba480a803a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:15.868306 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bdf8e840-5c0d-4ce5-a964-60ba480a803a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:15.873488 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1053/2084] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:15 2022] DELETE /identity/v3/projects/83c45254f5f4416eb1bc3bbba7242e4b => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:15.874720 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35da810-4898-4fcc-aa6b-a7058f58ef0a None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:15.875009 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35da810-4898-4fcc-aa6b-a7058f58ef0a None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:15.875303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35da810-4898-4fcc-aa6b-a7058f58ef0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:15.900938 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-f35da810-4898-4fcc-aa6b-a7058f58ef0a None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:15.902260 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f35da810-4898-4fcc-aa6b-a7058f58ef0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:15.908544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4608ea-ad66-4e6a-8a13-3ee39b17a809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:15.909465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4608ea-ad66-4e6a-8a13-3ee39b17a809 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:15.909746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4608ea-ad66-4e6a-8a13-3ee39b17a809 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:15.910143 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4608ea-ad66-4e6a-8a13-3ee39b17a809 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:15.911521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4608ea-ad66-4e6a-8a13-3ee39b17a809 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:15.912542 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1054/2085] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:15.913503 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35da810-4898-4fcc-aa6b-a7058f58ef0a None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:15.913644 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4608ea-ad66-4e6a-8a13-3ee39b17a809 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:15.915361 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f35da810-4898-4fcc-aa6b-a7058f58ef0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:15.929282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1032/2086] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:15 2022] GET /identity/v3/roles => generated 1420 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:15.929282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be276981-5ebd-406b-87f3-3afd81b7b986 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:15.929282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be276981-5ebd-406b-87f3-3afd81b7b986 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:15.929282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be276981-5ebd-406b-87f3-3afd81b7b986 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:15.932333 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35da810-4898-4fcc-aa6b-a7058f58ef0a None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:15.939860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643b4899-2efd-493d-afe6-576f4c3118e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:15.940815 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643b4899-2efd-493d-afe6-576f4c3118e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:15.941748 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643b4899-2efd-493d-afe6-576f4c3118e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:15.942134 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643b4899-2efd-493d-afe6-576f4c3118e5 admin admin] PATH_INFO: `/v3/projects/91a5bb68aa1a4c4abe6ad691852e9dfe` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:15.946224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643b4899-2efd-493d-afe6-576f4c3118e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=91a5bb68aa1a4c4abe6ad691852e9dfe)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:15.946224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643b4899-2efd-493d-afe6-576f4c3118e5 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:15.970794 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-be276981-5ebd-406b-87f3-3afd81b7b986 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:15.972098 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-be276981-5ebd-406b-87f3-3afd81b7b986 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:15.980613 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1033/2087] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:15.981548 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be276981-5ebd-406b-87f3-3afd81b7b986 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:15.983003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-be276981-5ebd-406b-87f3-3afd81b7b986 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:16.001422 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be276981-5ebd-406b-87f3-3afd81b7b986 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:16.007845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-643b4899-2efd-493d-afe6-576f4c3118e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '91a5bb68aa1a4c4abe6ad691852e9dfe', 'request_id': 'req-643b4899-2efd-493d-afe6-576f4c3118e5'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:16.015931 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-643b4899-2efd-493d-afe6-576f4c3118e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '91a5bb68aa1a4c4abe6ad691852e9dfe', 'request_id': 'req-643b4899-2efd-493d-afe6-576f4c3118e5'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:16.027568 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-643b4899-2efd-493d-afe6-576f4c3118e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '91a5bb68aa1a4c4abe6ad691852e9dfe', 'request_id': 'req-643b4899-2efd-493d-afe6-576f4c3118e5'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:16.043406 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-643b4899-2efd-493d-afe6-576f4c3118e5 admin admin] The token cache is being invalidate because project 91a5bb68aa1a4c4abe6ad691852e9dfe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:16.043712 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-643b4899-2efd-493d-afe6-576f4c3118e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:16.044566 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-643b4899-2efd-493d-afe6-576f4c3118e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:16.050412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1055/2088] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:15 2022] DELETE /identity/v3/projects/91a5bb68aa1a4c4abe6ad691852e9dfe => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:16.050412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643b4899-2efd-493d-afe6-576f4c3118e5 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:16.050412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-643b4899-2efd-493d-afe6-576f4c3118e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:16.101158 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1c0b40-63de-4a9e-aefa-645f73f5a8bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:16.101902 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1c0b40-63de-4a9e-aefa-645f73f5a8bb admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.102139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1c0b40-63de-4a9e-aefa-645f73f5a8bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.102413 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1c0b40-63de-4a9e-aefa-645f73f5a8bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.103060 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1c0b40-63de-4a9e-aefa-645f73f5a8bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:16.104656 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1c0b40-63de-4a9e-aefa-645f73f5a8bb admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:16.118565 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1034/2089] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:15 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:16.118565 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e905d5-f041-4368-98f4-ed22869e7c71 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.118565 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e905d5-f041-4368-98f4-ed22869e7c71 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.118565 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e905d5-f041-4368-98f4-ed22869e7c71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.134728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643b4899-2efd-493d-afe6-576f4c3118e5 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:16.140443 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8820bd-a126-4340-b0b2-3de4d0711b5e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:16.141951 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8820bd-a126-4340-b0b2-3de4d0711b5e service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.142988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8820bd-a126-4340-b0b2-3de4d0711b5e service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.144257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-82e905d5-f041-4368-98f4-ed22869e7c71 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:16.144866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8820bd-a126-4340-b0b2-3de4d0711b5e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.145520 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-82e905d5-f041-4368-98f4-ed22869e7c71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:16.147859 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-9f8820bd-a126-4340-b0b2-3de4d0711b5e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:16.156860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1035/2090] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:16.156860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3e866d-716a-4d41-b7c7-8d0bdc77722b None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.156860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3e866d-716a-4d41-b7c7-8d0bdc77722b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.156860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3e866d-716a-4d41-b7c7-8d0bdc77722b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.176335 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8820bd-a126-4340-b0b2-3de4d0711b5e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:16.178444 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-be3e866d-716a-4d41-b7c7-8d0bdc77722b None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:16.178765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8820bd-a126-4340-b0b2-3de4d0711b5e service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:16.189578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-be3e866d-716a-4d41-b7c7-8d0bdc77722b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:16.189578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1036/2091] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:16.190221 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3e866d-716a-4d41-b7c7-8d0bdc77722b None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:16.191466 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-be3e866d-716a-4d41-b7c7-8d0bdc77722b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:16.204466 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1056/2092] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:16.204665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3e866d-716a-4d41-b7c7-8d0bdc77722b None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:16.205572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8820bd-a126-4340-b0b2-3de4d0711b5e service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:16.209889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f39d1da-ec2d-4e58-9007-57501d6c9221 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:16.210674 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f39d1da-ec2d-4e58-9007-57501d6c9221 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.211090 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f39d1da-ec2d-4e58-9007-57501d6c9221 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.211295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f39d1da-ec2d-4e58-9007-57501d6c9221 admin admin] PATH_INFO: `/v3/projects/b3c661e1e7d1467884fbc66821050373/users/46dc393dda754d6fad0471ba329638c8/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.221496 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8820bd-a126-4340-b0b2-3de4d0711b5e service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:16.230780 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bd0358-a3f1-46df-8949-fdf3664b8794 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:16.231327 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f39d1da-ec2d-4e58-9007-57501d6c9221 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3c661e1e7d1467884fbc66821050373, user_id=46dc393dda754d6fad0471ba329638c8, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:16.231567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bd0358-a3f1-46df-8949-fdf3664b8794 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.231949 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bd0358-a3f1-46df-8949-fdf3664b8794 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.232505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bd0358-a3f1-46df-8949-fdf3664b8794 admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.233431 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bd0358-a3f1-46df-8949-fdf3664b8794 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:27:16.233773 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bd0358-a3f1-46df-8949-fdf3664b8794 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:16.235822 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f39d1da-ec2d-4e58-9007-57501d6c9221 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:16.235984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bd0358-a3f1-46df-8949-fdf3664b8794 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:16.253364 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1057/2093] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:27:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:27:16.254037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1037/2094] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:16 2022] PUT /identity/v3/projects/b3c661e1e7d1467884fbc66821050373/users/46dc393dda754d6fad0471ba329638c8/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:16.255356 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d933ad-1365-48fc-be6d-c579f34b9b4f None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.255538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d933ad-1365-48fc-be6d-c579f34b9b4f None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.255835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d933ad-1365-48fc-be6d-c579f34b9b4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.257503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f39d1da-ec2d-4e58-9007-57501d6c9221 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:16.279819 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-10d933ad-1365-48fc-be6d-c579f34b9b4f None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:16.281866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-10d933ad-1365-48fc-be6d-c579f34b9b4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:16.333343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:16.356182 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f39d1da-ec2d-4e58-9007-57501d6c9221 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:16.358703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:16.365002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e2ed51-abc1-4565-adbb-49b83ebba3b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:16.365869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e2ed51-abc1-4565-adbb-49b83ebba3b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.366229 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e2ed51-abc1-4565-adbb-49b83ebba3b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.366525 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e2ed51-abc1-4565-adbb-49b83ebba3b1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.367397 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e2ed51-abc1-4565-adbb-49b83ebba3b1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:16.373231 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e2ed51-abc1-4565-adbb-49b83ebba3b1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:16.383007 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1058/2095] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:16.384591 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9547be35-5b99-47eb-98d8-c74a3dcc1e30 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.385131 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9547be35-5b99-47eb-98d8-c74a3dcc1e30 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.385842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9547be35-5b99-47eb-98d8-c74a3dcc1e30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.410868 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1038/2096] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:16 2022] POST /identity/v3/projects => generated 373 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:16.410868 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e2ed51-abc1-4565-adbb-49b83ebba3b1 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:16.447155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-9547be35-5b99-47eb-98d8-c74a3dcc1e30 None None] MFA Rules not processed for user `46dc393dda754d6fad0471ba329638c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:16.449931 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-9547be35-5b99-47eb-98d8-c74a3dcc1e30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:16.507762 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e2ed51-abc1-4565-adbb-49b83ebba3b1 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:16.515475 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e7bfdf-0285-46f7-9562-a865f0783ff8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:16.515691 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e7bfdf-0285-46f7-9562-a865f0783ff8 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.516013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e7bfdf-0285-46f7-9562-a865f0783ff8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.516645 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e7bfdf-0285-46f7-9562-a865f0783ff8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.517737 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e7bfdf-0285-46f7-9562-a865f0783ff8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:27:16.518330 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e7bfdf-0285-46f7-9562-a865f0783ff8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:16.520196 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e7bfdf-0285-46f7-9562-a865f0783ff8 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:16.536013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1039/2097] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:27:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:27:16.537122 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e7bfdf-0285-46f7-9562-a865f0783ff8 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:16.552239 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e7bfdf-0285-46f7-9562-a865f0783ff8 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:16.558727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0847f7a9-f866-4371-8def-595c44dfe7bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:16.559162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0847f7a9-f866-4371-8def-595c44dfe7bf admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.559445 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0847f7a9-f866-4371-8def-595c44dfe7bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.559781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0847f7a9-f866-4371-8def-595c44dfe7bf admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.560494 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0847f7a9-f866-4371-8def-595c44dfe7bf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:16.562845 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0847f7a9-f866-4371-8def-595c44dfe7bf admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:16.575159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1059/2098] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:16 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:16.576288 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9547be35-5b99-47eb-98d8-c74a3dcc1e30 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:16.591480 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1040/2099] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:16 2022] POST /identity/v3/users => generated 287 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:16.592481 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0847f7a9-f866-4371-8def-595c44dfe7bf admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:16.596352 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9547be35-5b99-47eb-98d8-c74a3dcc1e30 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:16.607854 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0847f7a9-f866-4371-8def-595c44dfe7bf admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:16.608044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d4ae3f-eb3f-4725-8204-ae908ea8cde4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:16.608727 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d4ae3f-eb3f-4725-8204-ae908ea8cde4 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.609100 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d4ae3f-eb3f-4725-8204-ae908ea8cde4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.609501 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d4ae3f-eb3f-4725-8204-ae908ea8cde4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.610198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d4ae3f-eb3f-4725-8204-ae908ea8cde4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:16.612548 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d4ae3f-eb3f-4725-8204-ae908ea8cde4 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:16.613932 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a694e127-5652-4ea5-bff1-e22a73cbdfa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:16.614625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a694e127-5652-4ea5-bff1-e22a73cbdfa7 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.614979 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a694e127-5652-4ea5-bff1-e22a73cbdfa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.615380 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a694e127-5652-4ea5-bff1-e22a73cbdfa7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.616177 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a694e127-5652-4ea5-bff1-e22a73cbdfa7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:16.618449 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a694e127-5652-4ea5-bff1-e22a73cbdfa7 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:16.658706 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1060/2100] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:16 2022] POST /identity/v3/projects => generated 377 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:16.659086 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1041/2101] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:16 2022] POST /identity/v3/projects => generated 393 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:16.660101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a694e127-5652-4ea5-bff1-e22a73cbdfa7 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:16.660489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d4ae3f-eb3f-4725-8204-ae908ea8cde4 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:16.774215 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d4ae3f-eb3f-4725-8204-ae908ea8cde4 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:16.781156 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299b99e8-3a90-445a-b5e6-34a8715c32a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:16.782007 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299b99e8-3a90-445a-b5e6-34a8715c32a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.782445 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299b99e8-3a90-445a-b5e6-34a8715c32a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.785128 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299b99e8-3a90-445a-b5e6-34a8715c32a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.789416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299b99e8-3a90-445a-b5e6-34a8715c32a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:16.791901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a694e127-5652-4ea5-bff1-e22a73cbdfa7 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:16.792048 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299b99e8-3a90-445a-b5e6-34a8715c32a0 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:16.807543 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547d681f-3259-4dae-8dba-1604f9f1ec94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:16.808181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547d681f-3259-4dae-8dba-1604f9f1ec94 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.808459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547d681f-3259-4dae-8dba-1604f9f1ec94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.808727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547d681f-3259-4dae-8dba-1604f9f1ec94 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.809962 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547d681f-3259-4dae-8dba-1604f9f1ec94 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:16.811899 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547d681f-3259-4dae-8dba-1604f9f1ec94 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:16.825285 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1061/2102] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:16 2022] GET /identity/v3/roles => generated 1420 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:16.825285 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299b99e8-3a90-445a-b5e6-34a8715c32a0 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:16.839346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299b99e8-3a90-445a-b5e6-34a8715c32a0 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:16.848952 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199209f3-0000-4a9d-aea7-5441f05d1e7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:16.849816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1042/2103] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:16 2022] POST /identity/v3/users => generated 303 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:16.850771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547d681f-3259-4dae-8dba-1604f9f1ec94 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:16.852689 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199209f3-0000-4a9d-aea7-5441f05d1e7e admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.853169 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199209f3-0000-4a9d-aea7-5441f05d1e7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.853561 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199209f3-0000-4a9d-aea7-5441f05d1e7e admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.854335 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199209f3-0000-4a9d-aea7-5441f05d1e7e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:16.856424 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199209f3-0000-4a9d-aea7-5441f05d1e7e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:16.863319 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547d681f-3259-4dae-8dba-1604f9f1ec94 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:16.870559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a068287b-571e-4333-a3a0-cef4b0943af6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:16.871268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a068287b-571e-4333-a3a0-cef4b0943af6 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.871558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a068287b-571e-4333-a3a0-cef4b0943af6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.871828 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a068287b-571e-4333-a3a0-cef4b0943af6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.872708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a068287b-571e-4333-a3a0-cef4b0943af6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:16.874526 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a068287b-571e-4333-a3a0-cef4b0943af6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:16.896927 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1062/2104] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:16 2022] POST /identity/v3/users => generated 289 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:16.897827 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199209f3-0000-4a9d-aea7-5441f05d1e7e admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:16.903478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1043/2105] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:16 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:16.904458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a068287b-571e-4333-a3a0-cef4b0943af6 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:16.912806 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199209f3-0000-4a9d-aea7-5441f05d1e7e admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:16.918630 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a068287b-571e-4333-a3a0-cef4b0943af6 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:16.919505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e864b9-84ad-40b2-8b3d-d590679d1e55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:16.920259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e864b9-84ad-40b2-8b3d-d590679d1e55 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.920553 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e864b9-84ad-40b2-8b3d-d590679d1e55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.920946 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e864b9-84ad-40b2-8b3d-d590679d1e55 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.921772 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e864b9-84ad-40b2-8b3d-d590679d1e55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:16.924120 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e864b9-84ad-40b2-8b3d-d590679d1e55 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:16.925433 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb4ebe2-dad2-4d32-b123-7ab738be65d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:16.925706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb4ebe2-dad2-4d32-b123-7ab738be65d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.926103 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb4ebe2-dad2-4d32-b123-7ab738be65d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.927032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb4ebe2-dad2-4d32-b123-7ab738be65d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.928020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb4ebe2-dad2-4d32-b123-7ab738be65d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:16.929665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb4ebe2-dad2-4d32-b123-7ab738be65d4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:16.944938 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1063/2106] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:16 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:16.945873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e864b9-84ad-40b2-8b3d-d590679d1e55 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:16.950133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1044/2107] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:16 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:16.954868 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb4ebe2-dad2-4d32-b123-7ab738be65d4 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:16.958228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e864b9-84ad-40b2-8b3d-d590679d1e55 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:16.964240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:16.969252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b265026c-3793-427f-b889-44d030b73e2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:16.969943 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b265026c-3793-427f-b889-44d030b73e2b admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.970796 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b265026c-3793-427f-b889-44d030b73e2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.971367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b265026c-3793-427f-b889-44d030b73e2b admin admin] PATH_INFO: `/v3/projects/c14d67d1995e43b6908df25fc429f8ad/users/960a948161444298a3c4cc687fbc1c47/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:16.977701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb4ebe2-dad2-4d32-b123-7ab738be65d4 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:16.996097 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22b102b-4896-4540-9e06-12c13898e302 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:16.996790 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22b102b-4896-4540-9e06-12c13898e302 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:16.997093 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22b102b-4896-4540-9e06-12c13898e302 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:16.997381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22b102b-4896-4540-9e06-12c13898e302 admin admin] PATH_INFO: `/v3/projects/05a904572665490e947571f5d80555d1/users/85b0e654512043708b06ebefe55a3fb6/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:17.010875 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b265026c-3793-427f-b889-44d030b73e2b admin admin] RBAC: Authorizing `identity:create_grant(project_id=c14d67d1995e43b6908df25fc429f8ad, user_id=960a948161444298a3c4cc687fbc1c47, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:17.013640 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b265026c-3793-427f-b889-44d030b73e2b admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:17.028699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22b102b-4896-4540-9e06-12c13898e302 admin admin] RBAC: Authorizing `identity:create_grant(project_id=05a904572665490e947571f5d80555d1, user_id=85b0e654512043708b06ebefe55a3fb6, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:17.040863 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22b102b-4896-4540-9e06-12c13898e302 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:17.042478 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1064/2108] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:16 2022] PUT /identity/v3/projects/c14d67d1995e43b6908df25fc429f8ad/users/960a948161444298a3c4cc687fbc1c47/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:17.065309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b265026c-3793-427f-b889-44d030b73e2b admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:17.076048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1045/2109] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:16 2022] PUT /identity/v3/projects/05a904572665490e947571f5d80555d1/users/85b0e654512043708b06ebefe55a3fb6/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:17.077921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b667fae-e107-4d64-a1de-798685c01752 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:17.077921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b667fae-e107-4d64-a1de-798685c01752 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:17.078465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b667fae-e107-4d64-a1de-798685c01752 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:17.089923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:17.127716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-9b667fae-e107-4d64-a1de-798685c01752 None None] MFA Rules not processed for user `960a948161444298a3c4cc687fbc1c47`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:17.129347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9b667fae-e107-4d64-a1de-798685c01752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:17.175418 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b265026c-3793-427f-b889-44d030b73e2b admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:17.234249 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1046/2110] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:17 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:17.234249 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b667fae-e107-4d64-a1de-798685c01752 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:17.281242 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf264e4-01bc-4eb9-a3ee-79ac0cd85dc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:17.281856 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf264e4-01bc-4eb9-a3ee-79ac0cd85dc2 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:17.283617 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf264e4-01bc-4eb9-a3ee-79ac0cd85dc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:17.284465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf264e4-01bc-4eb9-a3ee-79ac0cd85dc2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:17.285193 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf264e4-01bc-4eb9-a3ee-79ac0cd85dc2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:17.286868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf264e4-01bc-4eb9-a3ee-79ac0cd85dc2 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:17.324755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1065/2111] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:17 2022] GET /identity/v3/roles => generated 1420 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:17.326654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf264e4-01bc-4eb9-a3ee-79ac0cd85dc2 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:17.331162 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-fcf264e4-01bc-4eb9-a3ee-79ac0cd85dc2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:17.339109 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b667fae-e107-4d64-a1de-798685c01752 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:17.352008 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54c2d3d-11d9-4515-ba5b-6e0bf4e668b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:17.354533 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54c2d3d-11d9-4515-ba5b-6e0bf4e668b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:17.355372 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54c2d3d-11d9-4515-ba5b-6e0bf4e668b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:17.356365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54c2d3d-11d9-4515-ba5b-6e0bf4e668b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:17.358035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54c2d3d-11d9-4515-ba5b-6e0bf4e668b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:17.360611 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54c2d3d-11d9-4515-ba5b-6e0bf4e668b6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:17.405937 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1047/2112] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:17 2022] GET /identity/v3/roles => generated 1420 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:17.407391 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54c2d3d-11d9-4515-ba5b-6e0bf4e668b6 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:17.422772 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf264e4-01bc-4eb9-a3ee-79ac0cd85dc2 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:17.424550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54c2d3d-11d9-4515-ba5b-6e0bf4e668b6 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:17.429798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616a1cc7-e4b4-457f-9cd9-7a3dfee66aa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:17.430463 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616a1cc7-e4b4-457f-9cd9-7a3dfee66aa7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:17.430757 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616a1cc7-e4b4-457f-9cd9-7a3dfee66aa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:17.431060 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616a1cc7-e4b4-457f-9cd9-7a3dfee66aa7 admin admin] PATH_INFO: `/v3/projects/a58ceed104724b9193d4a2f6f1ff822b/users/80a36eeae4914e5ca58c5ce538226896/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:17.434439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84febb82-a9c9-42d6-9c41-f471bba8b71b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:17.435117 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84febb82-a9c9-42d6-9c41-f471bba8b71b service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:17.435394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84febb82-a9c9-42d6-9c41-f471bba8b71b service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:17.435669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84febb82-a9c9-42d6-9c41-f471bba8b71b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:17.446645 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84febb82-a9c9-42d6-9c41-f471bba8b71b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:17.448112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84febb82-a9c9-42d6-9c41-f471bba8b71b service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:17.453533 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616a1cc7-e4b4-457f-9cd9-7a3dfee66aa7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a58ceed104724b9193d4a2f6f1ff822b, user_id=80a36eeae4914e5ca58c5ce538226896, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:17.457199 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616a1cc7-e4b4-457f-9cd9-7a3dfee66aa7 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:17.458357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1066/2113] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:17 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:17.468473 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84febb82-a9c9-42d6-9c41-f471bba8b71b service cinder] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:17.494267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84febb82-a9c9-42d6-9c41-f471bba8b71b service cinder] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:17.495344 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1048/2114] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:17 2022] PUT /identity/v3/projects/a58ceed104724b9193d4a2f6f1ff822b/users/80a36eeae4914e5ca58c5ce538226896/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:17.496923 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc76b1f7-dcab-45ea-8050-a70f9cabd3ac None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:17.497373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc76b1f7-dcab-45ea-8050-a70f9cabd3ac None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:17.497661 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc76b1f7-dcab-45ea-8050-a70f9cabd3ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:17.542189 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-bc76b1f7-dcab-45ea-8050-a70f9cabd3ac None None] MFA Rules not processed for user `3a25d172babb4cd698fac79d7412d25b`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:17.543882 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bc76b1f7-dcab-45ea-8050-a70f9cabd3ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:17.555875 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:17.568705 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520e1ec0-85a3-49ce-8146-ea42d6c33108 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:17.569517 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520e1ec0-85a3-49ce-8146-ea42d6c33108 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:17.570387 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520e1ec0-85a3-49ce-8146-ea42d6c33108 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:17.570723 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520e1ec0-85a3-49ce-8146-ea42d6c33108 admin admin] PATH_INFO: `/v3/projects/05a904572665490e947571f5d80555d1/users/85b0e654512043708b06ebefe55a3fb6/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:17.573146 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520e1ec0-85a3-49ce-8146-ea42d6c33108 admin admin] RBAC: Authorizing `identity:create_grant(project_id=05a904572665490e947571f5d80555d1, user_id=85b0e654512043708b06ebefe55a3fb6, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:17.576180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520e1ec0-85a3-49ce-8146-ea42d6c33108 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:17.583575 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:17.589997 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1067/2115] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:17 2022] PUT /identity/v3/projects/05a904572665490e947571f5d80555d1/users/85b0e654512043708b06ebefe55a3fb6/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:17.590766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520e1ec0-85a3-49ce-8146-ea42d6c33108 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:17.639278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1049/2116] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:17 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:17.644260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4284ac-6991-4b4a-9bd6-16607858faeb None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:17.644554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4284ac-6991-4b4a-9bd6-16607858faeb None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:17.644828 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4284ac-6991-4b4a-9bd6-16607858faeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:17.686423 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ae4284ac-6991-4b4a-9bd6-16607858faeb None None] MFA Rules not processed for user `80a36eeae4914e5ca58c5ce538226896`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:17.687820 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ae4284ac-6991-4b4a-9bd6-16607858faeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:17.692795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520e1ec0-85a3-49ce-8146-ea42d6c33108 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:17.699950 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7e14fb-5241-4a04-a15c-127140a815d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:17.700643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7e14fb-5241-4a04-a15c-127140a815d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:17.700924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7e14fb-5241-4a04-a15c-127140a815d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:17.701441 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7e14fb-5241-4a04-a15c-127140a815d2 admin admin] PATH_INFO: `/v3/users/960a948161444298a3c4cc687fbc1c47` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:17.703412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7e14fb-5241-4a04-a15c-127140a815d2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=960a948161444298a3c4cc687fbc1c47)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:17.706531 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7e14fb-5241-4a04-a15c-127140a815d2 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:17.788692 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-2e7e14fb-5241-4a04-a15c-127140a815d2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '960a948161444298a3c4cc687fbc1c47'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:17.794117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1050/2117] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:17 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:17.799469 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb63b09-1965-4949-aee9-0d44c71c08dd None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:17.799760 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb63b09-1965-4949-aee9-0d44c71c08dd None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:17.800055 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb63b09-1965-4949-aee9-0d44c71c08dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:17.803288 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-2e7e14fb-5241-4a04-a15c-127140a815d2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '960a948161444298a3c4cc687fbc1c47'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:17.803952 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-2e7e14fb-5241-4a04-a15c-127140a815d2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '960a948161444298a3c4cc687fbc1c47'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:17.822183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-2e7e14fb-5241-4a04-a15c-127140a815d2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '960a948161444298a3c4cc687fbc1c47'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:17.823593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-2e7e14fb-5241-4a04-a15c-127140a815d2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '960a948161444298a3c4cc687fbc1c47'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:17.842376 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1068/2118] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:17 2022] DELETE /identity/v3/users/960a948161444298a3c4cc687fbc1c47 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:17.843111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7e14fb-5241-4a04-a15c-127140a815d2 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:17.845857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-2e7e14fb-5241-4a04-a15c-127140a815d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:17.849432 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-0cb63b09-1965-4949-aee9-0d44c71c08dd None None] MFA Rules not processed for user `85b0e654512043708b06ebefe55a3fb6`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:17.850951 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0cb63b09-1965-4949-aee9-0d44c71c08dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:17.939203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7e14fb-5241-4a04-a15c-127140a815d2 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:17.948042 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b54fa40-c2dd-4bf9-8b4a-f3cdbc4cc33a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:17.949025 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b54fa40-c2dd-4bf9-8b4a-f3cdbc4cc33a service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:17.949025 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b54fa40-c2dd-4bf9-8b4a-f3cdbc4cc33a service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:17.949272 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b54fa40-c2dd-4bf9-8b4a-f3cdbc4cc33a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:17.951036 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4b54fa40-c2dd-4bf9-8b4a-f3cdbc4cc33a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:17.967636 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1051/2119] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:17 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:17.968271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb63b09-1965-4949-aee9-0d44c71c08dd None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:17.969834 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0cb63b09-1965-4949-aee9-0d44c71c08dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:17.984891 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:18.036342 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:18.043259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b54fa40-c2dd-4bf9-8b4a-f3cdbc4cc33a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:18.044729 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b54fa40-c2dd-4bf9-8b4a-f3cdbc4cc33a service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:18.054505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1069/2120] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:17 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 212 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:18.056390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9b8eed-85e6-4e8f-98bb-43ad27de4495 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:18.057181 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9b8eed-85e6-4e8f-98bb-43ad27de4495 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:18.057181 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9b8eed-85e6-4e8f-98bb-43ad27de4495 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:18.063544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb63b09-1965-4949-aee9-0d44c71c08dd None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:18.071682 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8442b15-f9dc-49ee-af29-41ab7be7b797 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:18.071834 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8442b15-f9dc-49ee-af29-41ab7be7b797 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:18.072111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8442b15-f9dc-49ee-af29-41ab7be7b797 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:18.072461 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8442b15-f9dc-49ee-af29-41ab7be7b797 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:18.073399 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8442b15-f9dc-49ee-af29-41ab7be7b797 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:18.075412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8442b15-f9dc-49ee-af29-41ab7be7b797 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:18.081241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-3b9b8eed-85e6-4e8f-98bb-43ad27de4495 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:18.084035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-3b9b8eed-85e6-4e8f-98bb-43ad27de4495 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:18.096072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1070/2121] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:18.096072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9b8eed-85e6-4e8f-98bb-43ad27de4495 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:18.096072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-3b9b8eed-85e6-4e8f-98bb-43ad27de4495 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:18.117551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9b8eed-85e6-4e8f-98bb-43ad27de4495 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:18.128714 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1052/2122] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:17 2022] POST /identity/v3/projects => generated 377 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:18.129591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8442b15-f9dc-49ee-af29-41ab7be7b797 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:18.130365 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a27a0b9-1409-42a7-a423-d4518999f514 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:18.131149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a27a0b9-1409-42a7-a423-d4518999f514 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:18.131454 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a27a0b9-1409-42a7-a423-d4518999f514 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:18.132361 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a27a0b9-1409-42a7-a423-d4518999f514 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:18.134984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f8442b15-f9dc-49ee-af29-41ab7be7b797 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:18.150654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a27a0b9-1409-42a7-a423-d4518999f514 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:18.152579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a27a0b9-1409-42a7-a423-d4518999f514 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:18.189860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1071/2123] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:18 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:18.190749 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a27a0b9-1409-42a7-a423-d4518999f514 service cinder] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:18.212839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a27a0b9-1409-42a7-a423-d4518999f514 service cinder] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:18.246467 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8442b15-f9dc-49ee-af29-41ab7be7b797 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:18.258518 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8266d689-101d-4aa7-9aba-7af97381a05b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:18.258814 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8266d689-101d-4aa7-9aba-7af97381a05b service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:18.259123 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8266d689-101d-4aa7-9aba-7af97381a05b service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:18.259395 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8266d689-101d-4aa7-9aba-7af97381a05b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:18.366427 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1170a5c4-5b15-4255-8ab9-4d1c8bcdf8ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:18.367100 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1170a5c4-5b15-4255-8ab9-4d1c8bcdf8ce admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:18.367447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1170a5c4-5b15-4255-8ab9-4d1c8bcdf8ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:18.367815 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1170a5c4-5b15-4255-8ab9-4d1c8bcdf8ce admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:18.368880 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1170a5c4-5b15-4255-8ab9-4d1c8bcdf8ce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:18.371404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1170a5c4-5b15-4255-8ab9-4d1c8bcdf8ce admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:18.413697 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1072/2124] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:18 2022] POST /identity/v3/users => generated 289 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:18.414648 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1170a5c4-5b15-4255-8ab9-4d1c8bcdf8ce admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:18.423292 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8266d689-101d-4aa7-9aba-7af97381a05b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:18.425817 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8266d689-101d-4aa7-9aba-7af97381a05b service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:18.438215 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1053/2125] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:18.440219 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8266d689-101d-4aa7-9aba-7af97381a05b service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:18.464485 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8266d689-101d-4aa7-9aba-7af97381a05b service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:18.482793 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c80f9dc-1ffb-4dd7-aa2a-5c699891d776 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:18.486829 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c80f9dc-1ffb-4dd7-aa2a-5c699891d776 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:18.487295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c80f9dc-1ffb-4dd7-aa2a-5c699891d776 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:18.488166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c80f9dc-1ffb-4dd7-aa2a-5c699891d776 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:18.491013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c80f9dc-1ffb-4dd7-aa2a-5c699891d776 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:18.491572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1170a5c4-5b15-4255-8ab9-4d1c8bcdf8ce admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:18.495266 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c80f9dc-1ffb-4dd7-aa2a-5c699891d776 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:18.499044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f23b7f-2c98-43b5-9862-b51b2c5bff4f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:18.500032 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f23b7f-2c98-43b5-9862-b51b2c5bff4f service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:18.501120 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f23b7f-2c98-43b5-9862-b51b2c5bff4f service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:18.501675 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f23b7f-2c98-43b5-9862-b51b2c5bff4f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:18.504129 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d5f23b7f-2c98-43b5-9862-b51b2c5bff4f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:18.527244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1054/2126] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:18 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:18.532683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c80f9dc-1ffb-4dd7-aa2a-5c699891d776 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:18.562915 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c80f9dc-1ffb-4dd7-aa2a-5c699891d776 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:18.573549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54b409b-9a98-49d1-9bb0-29de204ae676 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:18.574641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54b409b-9a98-49d1-9bb0-29de204ae676 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:18.574915 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54b409b-9a98-49d1-9bb0-29de204ae676 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:18.575228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54b409b-9a98-49d1-9bb0-29de204ae676 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:18.575976 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54b409b-9a98-49d1-9bb0-29de204ae676 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:18.577680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54b409b-9a98-49d1-9bb0-29de204ae676 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:18.584420 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:18.593813 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1055/2127] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:18 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:18.598240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54b409b-9a98-49d1-9bb0-29de204ae676 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:18.614692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54b409b-9a98-49d1-9bb0-29de204ae676 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:18.633408 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bb770c-1051-4f2d-97e5-abfca5c8d467 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:18.633408 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bb770c-1051-4f2d-97e5-abfca5c8d467 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:18.633408 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bb770c-1051-4f2d-97e5-abfca5c8d467 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:18.633408 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bb770c-1051-4f2d-97e5-abfca5c8d467 admin admin] PATH_INFO: `/v3/projects/d3c6aa7d022a4c919cdf4b20184646b3/users/d2149700938d4df29d652d01029168b6/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:18.633759 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f23b7f-2c98-43b5-9862-b51b2c5bff4f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:18.633759 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f23b7f-2c98-43b5-9862-b51b2c5bff4f service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:18.659576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1073/2128] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:18 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:18.661802 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89bb770c-1051-4f2d-97e5-abfca5c8d467 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d3c6aa7d022a4c919cdf4b20184646b3, user_id=d2149700938d4df29d652d01029168b6, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:18.665021 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89bb770c-1051-4f2d-97e5-abfca5c8d467 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:18.692401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1056/2129] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:18 2022] PUT /identity/v3/projects/d3c6aa7d022a4c919cdf4b20184646b3/users/d2149700938d4df29d652d01029168b6/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:18.703462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd17879e-b3d3-4a6c-ba9a-ce5f9eee728a None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:18.703750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd17879e-b3d3-4a6c-ba9a-ce5f9eee728a None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:18.704026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd17879e-b3d3-4a6c-ba9a-ce5f9eee728a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:18.762261 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-bd17879e-b3d3-4a6c-ba9a-ce5f9eee728a None None] MFA Rules not processed for user `d2149700938d4df29d652d01029168b6`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:18.763769 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-bd17879e-b3d3-4a6c-ba9a-ce5f9eee728a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:18.858747 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bb770c-1051-4f2d-97e5-abfca5c8d467 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:18.863629 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-89bb770c-1051-4f2d-97e5-abfca5c8d467 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:18.890222 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1074/2130] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:18 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:18.892026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd17879e-b3d3-4a6c-ba9a-ce5f9eee728a None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:18.916341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:18.983659 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bb770c-1051-4f2d-97e5-abfca5c8d467 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:18.999949 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447aac42-43e7-40d5-83c6-7b804a0d1733 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:19.000690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447aac42-43e7-40d5-83c6-7b804a0d1733 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:19.001024 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447aac42-43e7-40d5-83c6-7b804a0d1733 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:19.001359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447aac42-43e7-40d5-83c6-7b804a0d1733 admin admin] PATH_INFO: `/v3/projects/c14d67d1995e43b6908df25fc429f8ad` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:19.004849 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447aac42-43e7-40d5-83c6-7b804a0d1733 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c14d67d1995e43b6908df25fc429f8ad)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:19.006820 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447aac42-43e7-40d5-83c6-7b804a0d1733 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:19.008164 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd17879e-b3d3-4a6c-ba9a-ce5f9eee728a None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:19.020213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06a5d78-48aa-4520-9690-fe4ce353e7da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:19.023575 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06a5d78-48aa-4520-9690-fe4ce353e7da admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:19.023870 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06a5d78-48aa-4520-9690-fe4ce353e7da admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:19.025297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06a5d78-48aa-4520-9690-fe4ce353e7da admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:19.025912 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06a5d78-48aa-4520-9690-fe4ce353e7da admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:19.031592 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06a5d78-48aa-4520-9690-fe4ce353e7da admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:19.128139 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1075/2131] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:18 2022] POST /identity/v3/projects => generated 377 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:19.129521 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06a5d78-48aa-4520-9690-fe4ce353e7da admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:19.162227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-447aac42-43e7-40d5-83c6-7b804a0d1733 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c14d67d1995e43b6908df25fc429f8ad', 'request_id': 'req-447aac42-43e7-40d5-83c6-7b804a0d1733'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:19.183243 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-447aac42-43e7-40d5-83c6-7b804a0d1733 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c14d67d1995e43b6908df25fc429f8ad', 'request_id': 'req-447aac42-43e7-40d5-83c6-7b804a0d1733'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:19.200754 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-447aac42-43e7-40d5-83c6-7b804a0d1733 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c14d67d1995e43b6908df25fc429f8ad', 'request_id': 'req-447aac42-43e7-40d5-83c6-7b804a0d1733'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:19.231002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-447aac42-43e7-40d5-83c6-7b804a0d1733 admin admin] The token cache is being invalidate because project c14d67d1995e43b6908df25fc429f8ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:19.231296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-447aac42-43e7-40d5-83c6-7b804a0d1733 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:19.232489 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-447aac42-43e7-40d5-83c6-7b804a0d1733 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:19.235302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1057/2132] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:18 2022] DELETE /identity/v3/projects/c14d67d1995e43b6908df25fc429f8ad => generated 0 bytes in 378 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:19.240039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae187c8-b373-4af8-a4d1-205fab99716c None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:19.241390 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae187c8-b373-4af8-a4d1-205fab99716c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:19.241861 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae187c8-b373-4af8-a4d1-205fab99716c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:19.270436 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3ae187c8-b373-4af8-a4d1-205fab99716c None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:19.272172 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3ae187c8-b373-4af8-a4d1-205fab99716c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:19.279533 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06a5d78-48aa-4520-9690-fe4ce353e7da admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:19.286116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413e07b1-9640-413a-87a0-459862afa676 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:19.289261 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413e07b1-9640-413a-87a0-459862afa676 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:19.289261 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413e07b1-9640-413a-87a0-459862afa676 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:19.289261 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413e07b1-9640-413a-87a0-459862afa676 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:19.289261 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413e07b1-9640-413a-87a0-459862afa676 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:19.290671 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413e07b1-9640-413a-87a0-459862afa676 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:19.301280 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:19.301518 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1058/2133] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:27:19.302977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bca81fd-54b8-4e7a-bd92-9b901184703d None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:19.303270 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bca81fd-54b8-4e7a-bd92-9b901184703d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:19.303542 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bca81fd-54b8-4e7a-bd92-9b901184703d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:19.323988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1076/2134] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:19 2022] POST /identity/v3/users => generated 295 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:19.328465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413e07b1-9640-413a-87a0-459862afa676 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:19.331456 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-413e07b1-9640-413a-87a0-459862afa676 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:19.344097 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-6bca81fd-54b8-4e7a-bd92-9b901184703d None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:19.345398 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6bca81fd-54b8-4e7a-bd92-9b901184703d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:19.352060 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413e07b1-9640-413a-87a0-459862afa676 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:19.355359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1059/2135] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:19.359716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bca81fd-54b8-4e7a-bd92-9b901184703d None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:19.363151 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a10e88-504b-4d66-b8b8-32d179392828 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:19.363965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a10e88-504b-4d66-b8b8-32d179392828 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:19.364251 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a10e88-504b-4d66-b8b8-32d179392828 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:19.365010 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a10e88-504b-4d66-b8b8-32d179392828 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:19.366081 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a10e88-504b-4d66-b8b8-32d179392828 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:19.367340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a10e88-504b-4d66-b8b8-32d179392828 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:19.371491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bca81fd-54b8-4e7a-bd92-9b901184703d None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:19.377999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcac1eb-88ec-4695-9123-006c38929e5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:19.378724 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcac1eb-88ec-4695-9123-006c38929e5c admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:19.379082 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcac1eb-88ec-4695-9123-006c38929e5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:19.379453 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcac1eb-88ec-4695-9123-006c38929e5c admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:19.380200 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcac1eb-88ec-4695-9123-006c38929e5c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:27:19.380581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcac1eb-88ec-4695-9123-006c38929e5c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:19.382898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcac1eb-88ec-4695-9123-006c38929e5c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:19.396649 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1077/2136] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:19 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:27:19.401394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a10e88-504b-4d66-b8b8-32d179392828 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:19.401569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1060/2137] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:27:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:27:19.402408 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcac1eb-88ec-4695-9123-006c38929e5c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:19.415909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a10e88-504b-4d66-b8b8-32d179392828 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:19.424015 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcac1eb-88ec-4695-9123-006c38929e5c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:19.429147 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc76d768-6983-4976-8f63-c90b8f22b8e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:19.429828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc76d768-6983-4976-8f63-c90b8f22b8e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:19.430359 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc76d768-6983-4976-8f63-c90b8f22b8e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:19.431206 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc76d768-6983-4976-8f63-c90b8f22b8e3 admin admin] PATH_INFO: `/v3/projects/3bca1b8e1b1f49129aaa607f9afd2c33/users/649753f2c021443abd604b4f4f8ad45a/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:19.434975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36159623-e337-4ae7-9e15-70f08c31e31a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:19.435671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36159623-e337-4ae7-9e15-70f08c31e31a admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:19.436141 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36159623-e337-4ae7-9e15-70f08c31e31a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:19.436549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36159623-e337-4ae7-9e15-70f08c31e31a admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:19.437310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36159623-e337-4ae7-9e15-70f08c31e31a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:19.439900 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36159623-e337-4ae7-9e15-70f08c31e31a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:19.452134 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc76d768-6983-4976-8f63-c90b8f22b8e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3bca1b8e1b1f49129aaa607f9afd2c33, user_id=649753f2c021443abd604b4f4f8ad45a, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:19.456934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc76d768-6983-4976-8f63-c90b8f22b8e3 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:19.474999 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1078/2138] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:19 2022] PUT /identity/v3/projects/3bca1b8e1b1f49129aaa607f9afd2c33/users/649753f2c021443abd604b4f4f8ad45a/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:19.477954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1061/2139] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:19 2022] POST /identity/v3/projects => generated 383 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:19.478582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc76d768-6983-4976-8f63-c90b8f22b8e3 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:19.482473 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36159623-e337-4ae7-9e15-70f08c31e31a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:19.578228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc76d768-6983-4976-8f63-c90b8f22b8e3 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:19.588607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36159623-e337-4ae7-9e15-70f08c31e31a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:19.591357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab04559f-0b89-4e0d-b808-5f1622ccd7c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:19.594152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab04559f-0b89-4e0d-b808-5f1622ccd7c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:19.594499 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38a6973-7ee2-4140-a52e-1c85f8520807 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:19.594646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab04559f-0b89-4e0d-b808-5f1622ccd7c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:19.594918 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab04559f-0b89-4e0d-b808-5f1622ccd7c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:19.595119 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38a6973-7ee2-4140-a52e-1c85f8520807 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:19.595578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38a6973-7ee2-4140-a52e-1c85f8520807 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:19.595695 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab04559f-0b89-4e0d-b808-5f1622ccd7c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:19.595862 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38a6973-7ee2-4140-a52e-1c85f8520807 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:19.596580 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38a6973-7ee2-4140-a52e-1c85f8520807 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:19.598427 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab04559f-0b89-4e0d-b808-5f1622ccd7c3 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:19.599438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38a6973-7ee2-4140-a52e-1c85f8520807 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:19.632072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1079/2140] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:19 2022] GET /identity/v3/roles => generated 1420 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:19.633349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab04559f-0b89-4e0d-b808-5f1622ccd7c3 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:19.646861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab04559f-0b89-4e0d-b808-5f1622ccd7c3 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:19.655296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1062/2141] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:19 2022] POST /identity/v3/users => generated 292 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:19.660357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b236470a-d46c-4abf-b6b8-3967294ddde9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:19.660954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38a6973-7ee2-4140-a52e-1c85f8520807 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:19.662470 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b236470a-d46c-4abf-b6b8-3967294ddde9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:19.662980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b236470a-d46c-4abf-b6b8-3967294ddde9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:19.663511 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b236470a-d46c-4abf-b6b8-3967294ddde9 admin admin] PATH_INFO: `/v3/projects/3bca1b8e1b1f49129aaa607f9afd2c33/users/649753f2c021443abd604b4f4f8ad45a/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:19.666729 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b236470a-d46c-4abf-b6b8-3967294ddde9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3bca1b8e1b1f49129aaa607f9afd2c33, user_id=649753f2c021443abd604b4f4f8ad45a, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:19.668040 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:19.672374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b236470a-d46c-4abf-b6b8-3967294ddde9 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:19.672593 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38a6973-7ee2-4140-a52e-1c85f8520807 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:19.683038 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225731dc-8247-4604-a57b-e301281e99bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:19.683769 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225731dc-8247-4604-a57b-e301281e99bc admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:19.684600 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225731dc-8247-4604-a57b-e301281e99bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:19.684942 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225731dc-8247-4604-a57b-e301281e99bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:19.685764 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225731dc-8247-4604-a57b-e301281e99bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:19.687603 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225731dc-8247-4604-a57b-e301281e99bc admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:19.696651 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1080/2142] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:19 2022] PUT /identity/v3/projects/3bca1b8e1b1f49129aaa607f9afd2c33/users/649753f2c021443abd604b4f4f8ad45a/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:19.698293 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57940436-df1b-4449-b133-35d25c7a2b64 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:19.698658 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57940436-df1b-4449-b133-35d25c7a2b64 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:19.699005 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57940436-df1b-4449-b133-35d25c7a2b64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:19.706421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1063/2143] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:19 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:27:19.707490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225731dc-8247-4604-a57b-e301281e99bc admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:19.724375 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225731dc-8247-4604-a57b-e301281e99bc admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:19.733210 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-57940436-df1b-4449-b133-35d25c7a2b64 None None] MFA Rules not processed for user `649753f2c021443abd604b4f4f8ad45a`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:19.734646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-57940436-df1b-4449-b133-35d25c7a2b64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:19.825667 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594e9b27-96ff-4801-8ff1-94c9ec8bd4df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:19.826433 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594e9b27-96ff-4801-8ff1-94c9ec8bd4df admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:19.826715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594e9b27-96ff-4801-8ff1-94c9ec8bd4df admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:19.826994 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594e9b27-96ff-4801-8ff1-94c9ec8bd4df admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:19.828061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594e9b27-96ff-4801-8ff1-94c9ec8bd4df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:19.830004 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594e9b27-96ff-4801-8ff1-94c9ec8bd4df admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:19.850142 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1081/2144] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:19 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:19.851067 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1064/2145] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:19 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:27:19.852409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57940436-df1b-4449-b133-35d25c7a2b64 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:19.860140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594e9b27-96ff-4801-8ff1-94c9ec8bd4df admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:19.863421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-594e9b27-96ff-4801-8ff1-94c9ec8bd4df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:19.874814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57940436-df1b-4449-b133-35d25c7a2b64 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:19.881236 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e77853-8d25-4c33-b226-97127e58b6e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:19.882114 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e77853-8d25-4c33-b226-97127e58b6e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:19.882565 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e77853-8d25-4c33-b226-97127e58b6e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:19.883000 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e77853-8d25-4c33-b226-97127e58b6e7 admin admin] PATH_INFO: `/v3/projects/14d5c33e0f07495f933f33275c332822/users/789be65a6c804a24a17ad2d465754f4e/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:19.909866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e77853-8d25-4c33-b226-97127e58b6e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=14d5c33e0f07495f933f33275c332822, user_id=789be65a6c804a24a17ad2d465754f4e, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:19.912755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e77853-8d25-4c33-b226-97127e58b6e7 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:19.928735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:19.937003 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1082/2146] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:19 2022] PUT /identity/v3/projects/14d5c33e0f07495f933f33275c332822/users/789be65a6c804a24a17ad2d465754f4e/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:19.939062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc0cd15-7a5a-437f-9da5-e07fe7dc4e6d None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:19.940055 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc0cd15-7a5a-437f-9da5-e07fe7dc4e6d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:19.940304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc0cd15-7a5a-437f-9da5-e07fe7dc4e6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:19.965563 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-cbc0cd15-7a5a-437f-9da5-e07fe7dc4e6d None None] MFA Rules not processed for user `3a25d172babb4cd698fac79d7412d25b`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:19.969787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-cbc0cd15-7a5a-437f-9da5-e07fe7dc4e6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:19.987121 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594e9b27-96ff-4801-8ff1-94c9ec8bd4df admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:20.055275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1083/2147] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:19 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:20.056717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4a7093-7b70-4379-bebf-7e0b55be3061 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:20.057016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4a7093-7b70-4379-bebf-7e0b55be3061 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:20.057315 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4a7093-7b70-4379-bebf-7e0b55be3061 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:20.068568 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3cd0ae8-a702-45c2-9a0a-740c8c165a3e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:20.069437 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cd0ae8-a702-45c2-9a0a-740c8c165a3e service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:20.069852 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cd0ae8-a702-45c2-9a0a-740c8c165a3e service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:20.070209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cd0ae8-a702-45c2-9a0a-740c8c165a3e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:20.072361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c3cd0ae8-a702-45c2-9a0a-740c8c165a3e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:20.103602 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-eb4a7093-7b70-4379-bebf-7e0b55be3061 None None] MFA Rules not processed for user `789be65a6c804a24a17ad2d465754f4e`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:20.104862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-eb4a7093-7b70-4379-bebf-7e0b55be3061 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:20.173499 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3cd0ae8-a702-45c2-9a0a-740c8c165a3e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:20.176166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3cd0ae8-a702-45c2-9a0a-740c8c165a3e service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:20.183640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:20.203414 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1065/2148] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:19 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:20.254994 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1084/2149] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:20 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:20.259686 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3cd0ae8-a702-45c2-9a0a-740c8c165a3e service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:20.398343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3cd0ae8-a702-45c2-9a0a-740c8c165a3e service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:20.407872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4b3ecd-3f37-44f3-ae69-987035bcd27c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:20.408733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4b3ecd-3f37-44f3-ae69-987035bcd27c admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:20.409179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4b3ecd-3f37-44f3-ae69-987035bcd27c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:20.409487 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4b3ecd-3f37-44f3-ae69-987035bcd27c admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:20.410136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4b3ecd-3f37-44f3-ae69-987035bcd27c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:20.413156 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4b3ecd-3f37-44f3-ae69-987035bcd27c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:20.502551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1066/2150] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:20 2022] POST /identity/v3/projects => generated 385 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:20.506156 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4a7093-7b70-4379-bebf-7e0b55be3061 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:20.532516 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4a7093-7b70-4379-bebf-7e0b55be3061 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:20.628063 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69447ede-061f-4887-829e-53e461a4968e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:20.628783 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69447ede-061f-4887-829e-53e461a4968e admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:20.629199 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69447ede-061f-4887-829e-53e461a4968e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:20.629551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69447ede-061f-4887-829e-53e461a4968e admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:20.630295 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69447ede-061f-4887-829e-53e461a4968e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:20.633894 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69447ede-061f-4887-829e-53e461a4968e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:20.665018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1085/2151] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:20 2022] POST /identity/v3/users => generated 293 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:20.668680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4b3ecd-3f37-44f3-ae69-987035bcd27c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:20.684785 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4b3ecd-3f37-44f3-ae69-987035bcd27c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:20.691804 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e017fc5-1866-4e2d-9b88-0151447428a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:20.692990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e017fc5-1866-4e2d-9b88-0151447428a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:20.693756 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e017fc5-1866-4e2d-9b88-0151447428a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:20.694265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e017fc5-1866-4e2d-9b88-0151447428a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:20.695254 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e017fc5-1866-4e2d-9b88-0151447428a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:20.697636 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e017fc5-1866-4e2d-9b88-0151447428a2 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:20.711606 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1067/2152] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:20 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:20.715450 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69447ede-061f-4887-829e-53e461a4968e admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:20.732674 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69447ede-061f-4887-829e-53e461a4968e admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:20.741729 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4bf1be-51cc-45cf-a111-e1841d0e437a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:20.742583 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4bf1be-51cc-45cf-a111-e1841d0e437a admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:20.742903 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4bf1be-51cc-45cf-a111-e1841d0e437a admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:20.743278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4bf1be-51cc-45cf-a111-e1841d0e437a admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:20.745021 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4bf1be-51cc-45cf-a111-e1841d0e437a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:20.747574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4bf1be-51cc-45cf-a111-e1841d0e437a admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:20.766373 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1086/2153] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:20 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:20.770270 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e017fc5-1866-4e2d-9b88-0151447428a2 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:20.799214 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e017fc5-1866-4e2d-9b88-0151447428a2 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:20.818351 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771bcad9-5376-4edb-8f8e-0b13b7878766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:20.819124 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771bcad9-5376-4edb-8f8e-0b13b7878766 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:20.819413 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771bcad9-5376-4edb-8f8e-0b13b7878766 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:20.819692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771bcad9-5376-4edb-8f8e-0b13b7878766 admin admin] PATH_INFO: `/v3/projects/9d73542d9060479280ae58f5c51d2bcb/users/1779bef1781c40109927b7954da19b4a/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:20.855598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771bcad9-5376-4edb-8f8e-0b13b7878766 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9d73542d9060479280ae58f5c51d2bcb, user_id=1779bef1781c40109927b7954da19b4a, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:20.863969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771bcad9-5376-4edb-8f8e-0b13b7878766 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:20.908746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1068/2154] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:20 2022] PUT /identity/v3/projects/9d73542d9060479280ae58f5c51d2bcb/users/1779bef1781c40109927b7954da19b4a/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:20.910994 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfc87b3-90f5-4773-b605-a30a4dbbc154 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:20.911289 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfc87b3-90f5-4773-b605-a30a4dbbc154 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:20.911563 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfc87b3-90f5-4773-b605-a30a4dbbc154 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:20.998817 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:21.024777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-6cfc87b3-90f5-4773-b605-a30a4dbbc154 None None] MFA Rules not processed for user `1779bef1781c40109927b7954da19b4a`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:21.026480 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-6cfc87b3-90f5-4773-b605-a30a4dbbc154 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:21.141409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1087/2155] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:20 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:21.147188 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771bcad9-5376-4edb-8f8e-0b13b7878766 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:21.242874 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771bcad9-5376-4edb-8f8e-0b13b7878766 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:21.250100 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf1b001-9097-4890-b11b-0792fad8483f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:21.250754 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf1b001-9097-4890-b11b-0792fad8483f admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:21.251044 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf1b001-9097-4890-b11b-0792fad8483f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:21.251325 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf1b001-9097-4890-b11b-0792fad8483f admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:21.252495 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf1b001-9097-4890-b11b-0792fad8483f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:21.254465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf1b001-9097-4890-b11b-0792fad8483f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:21.296313 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfc87b3-90f5-4773-b605-a30a4dbbc154 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:21.297095 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1069/2156] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:21 2022] POST /identity/v3/projects => generated 385 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:21.313427 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfc87b3-90f5-4773-b605-a30a4dbbc154 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:21.417666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cad818c-22a6-45db-9b23-17b7db23a0b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:21.418688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cad818c-22a6-45db-9b23-17b7db23a0b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:21.419000 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cad818c-22a6-45db-9b23-17b7db23a0b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:21.419343 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cad818c-22a6-45db-9b23-17b7db23a0b2 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:21.420045 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cad818c-22a6-45db-9b23-17b7db23a0b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:21.422227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cad818c-22a6-45db-9b23-17b7db23a0b2 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:21.449110 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1088/2157] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:21 2022] POST /identity/v3/users => generated 299 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:27:21.453295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf1b001-9097-4890-b11b-0792fad8483f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:21.464385 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf1b001-9097-4890-b11b-0792fad8483f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:21.469429 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2382e11e-e5f3-424c-ad13-b27a1621dee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:21.470037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2382e11e-e5f3-424c-ad13-b27a1621dee2 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:21.470360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2382e11e-e5f3-424c-ad13-b27a1621dee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:21.470634 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2382e11e-e5f3-424c-ad13-b27a1621dee2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:21.471288 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2382e11e-e5f3-424c-ad13-b27a1621dee2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:21.472987 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2382e11e-e5f3-424c-ad13-b27a1621dee2 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:21.486555 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1070/2158] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:21 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:21.491860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cad818c-22a6-45db-9b23-17b7db23a0b2 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:21.503940 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cad818c-22a6-45db-9b23-17b7db23a0b2 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:21.508705 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db92a2a4-c7da-4f48-be6b-15cd390131b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:21.509395 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db92a2a4-c7da-4f48-be6b-15cd390131b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:21.509678 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db92a2a4-c7da-4f48-be6b-15cd390131b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:21.509951 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db92a2a4-c7da-4f48-be6b-15cd390131b9 admin admin] PATH_INFO: `/v3/projects/f510dd6017c44d3aaf001567f00cb76b/users/5981c254d4cf457cb8e3350e7ca12396/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:21.538934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db92a2a4-c7da-4f48-be6b-15cd390131b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f510dd6017c44d3aaf001567f00cb76b, user_id=5981c254d4cf457cb8e3350e7ca12396, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:21.542195 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db92a2a4-c7da-4f48-be6b-15cd390131b9 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:21.562902 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1089/2159] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:21 2022] PUT /identity/v3/projects/f510dd6017c44d3aaf001567f00cb76b/users/5981c254d4cf457cb8e3350e7ca12396/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:21.568801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2382e11e-e5f3-424c-ad13-b27a1621dee2 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:21.580310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2382e11e-e5f3-424c-ad13-b27a1621dee2 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:21.605867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:21.654951 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f1722b-4e8b-4009-a382-538b01c83b03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:21.655665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f1722b-4e8b-4009-a382-538b01c83b03 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:21.656450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f1722b-4e8b-4009-a382-538b01c83b03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:21.656726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f1722b-4e8b-4009-a382-538b01c83b03 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:21.657416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f1722b-4e8b-4009-a382-538b01c83b03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:21.659063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f1722b-4e8b-4009-a382-538b01c83b03 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:21.672229 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1071/2160] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:21 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:21.677436 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db92a2a4-c7da-4f48-be6b-15cd390131b9 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:21.690590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db92a2a4-c7da-4f48-be6b-15cd390131b9 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:21.695715 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1fff0b-9b3a-43e4-a9f7-66fcd100198a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:21.696444 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1fff0b-9b3a-43e4-a9f7-66fcd100198a admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:21.696835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1fff0b-9b3a-43e4-a9f7-66fcd100198a admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:21.697077 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1fff0b-9b3a-43e4-a9f7-66fcd100198a admin admin] PATH_INFO: `/v3/projects/f510dd6017c44d3aaf001567f00cb76b/users/5981c254d4cf457cb8e3350e7ca12396/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:21.699520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1fff0b-9b3a-43e4-a9f7-66fcd100198a admin admin] RBAC: Authorizing `identity:create_grant(project_id=f510dd6017c44d3aaf001567f00cb76b, user_id=5981c254d4cf457cb8e3350e7ca12396, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:21.702070 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1fff0b-9b3a-43e4-a9f7-66fcd100198a admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:21.713811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1090/2161] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:21 2022] PUT /identity/v3/projects/f510dd6017c44d3aaf001567f00cb76b/users/5981c254d4cf457cb8e3350e7ca12396/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:21.724370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c876c91-14d1-4dde-bd4a-42eaaa64447f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:21.724717 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c876c91-14d1-4dde-bd4a-42eaaa64447f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:21.724938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c876c91-14d1-4dde-bd4a-42eaaa64447f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:21.758359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-7c876c91-14d1-4dde-bd4a-42eaaa64447f None None] MFA Rules not processed for user `5981c254d4cf457cb8e3350e7ca12396`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:21.759672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7c876c91-14d1-4dde-bd4a-42eaaa64447f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:21.831034 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1072/2162] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:21 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:21.847244 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1fff0b-9b3a-43e4-a9f7-66fcd100198a admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:21.849193 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-cb1fff0b-9b3a-43e4-a9f7-66fcd100198a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:21.921204 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1fff0b-9b3a-43e4-a9f7-66fcd100198a admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:21.925674 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6641d5b-c4ef-4325-9136-1935c619a067 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:21.926280 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6641d5b-c4ef-4325-9136-1935c619a067 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:21.926572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6641d5b-c4ef-4325-9136-1935c619a067 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:21.926911 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6641d5b-c4ef-4325-9136-1935c619a067 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:21.933488 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:22.004193 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6641d5b-c4ef-4325-9136-1935c619a067 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:22.005708 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6641d5b-c4ef-4325-9136-1935c619a067 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:22.015159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c876c91-14d1-4dde-bd4a-42eaaa64447f None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:22.015492 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1091/2163] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:21 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:22.033471 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c876c91-14d1-4dde-bd4a-42eaaa64447f None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:22.038652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ee54bc-c5b7-4c9d-9655-3e4cf278e87a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:22.039376 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ee54bc-c5b7-4c9d-9655-3e4cf278e87a service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:22.039781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ee54bc-c5b7-4c9d-9655-3e4cf278e87a service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:22.040158 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ee54bc-c5b7-4c9d-9655-3e4cf278e87a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:22.042373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-71ee54bc-c5b7-4c9d-9655-3e4cf278e87a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:22.141073 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ee54bc-c5b7-4c9d-9655-3e4cf278e87a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:22.144863 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ee54bc-c5b7-4c9d-9655-3e4cf278e87a service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:22.156895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1073/2164] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:22 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:23.317055 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6641d5b-c4ef-4325-9136-1935c619a067 service cinder] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:23.326031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6641d5b-c4ef-4325-9136-1935c619a067 service cinder] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:23.329967 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd421a0-8735-4423-91c4-0fb13c47309c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:23.330575 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd421a0-8735-4423-91c4-0fb13c47309c service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:23.330854 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd421a0-8735-4423-91c4-0fb13c47309c service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:23.331127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd421a0-8735-4423-91c4-0fb13c47309c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:23.402752 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd421a0-8735-4423-91c4-0fb13c47309c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:23.404536 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd421a0-8735-4423-91c4-0fb13c47309c service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:23.422804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1092/2165] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:23 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:23.713702 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ee54bc-c5b7-4c9d-9655-3e4cf278e87a service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:23.723282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ee54bc-c5b7-4c9d-9655-3e4cf278e87a service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:23.727624 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c434ee7-0528-43ef-b140-7d592bfb145b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:23.728712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c434ee7-0528-43ef-b140-7d592bfb145b service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:23.729214 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c434ee7-0528-43ef-b140-7d592bfb145b service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:23.729500 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c434ee7-0528-43ef-b140-7d592bfb145b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:23.776164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:23.792235 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c434ee7-0528-43ef-b140-7d592bfb145b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:23.793794 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c434ee7-0528-43ef-b140-7d592bfb145b service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:23.804483 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1074/2166] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:23 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:24.043125 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd421a0-8735-4423-91c4-0fb13c47309c service cinder] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:24.048461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-6cd421a0-8735-4423-91c4-0fb13c47309c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:24.181852 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd421a0-8735-4423-91c4-0fb13c47309c service cinder] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:24.186905 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88a3c99-de25-4613-bd47-29c5e503fa92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:24.187766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88a3c99-de25-4613-bd47-29c5e503fa92 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:24.188049 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88a3c99-de25-4613-bd47-29c5e503fa92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:24.188317 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88a3c99-de25-4613-bd47-29c5e503fa92 admin admin] PATH_INFO: `/v3/users/46dc393dda754d6fad0471ba329638c8` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:24.190111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88a3c99-de25-4613-bd47-29c5e503fa92 admin admin] RBAC: Authorizing `identity:delete_user(user_id=46dc393dda754d6fad0471ba329638c8)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:24.192138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88a3c99-de25-4613-bd47-29c5e503fa92 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:24.247200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:24.253177 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b88a3c99-de25-4613-bd47-29c5e503fa92 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '46dc393dda754d6fad0471ba329638c8'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:24.262348 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b88a3c99-de25-4613-bd47-29c5e503fa92 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '46dc393dda754d6fad0471ba329638c8'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:24.263129 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b88a3c99-de25-4613-bd47-29c5e503fa92 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '46dc393dda754d6fad0471ba329638c8'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:24.281231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b88a3c99-de25-4613-bd47-29c5e503fa92 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '46dc393dda754d6fad0471ba329638c8'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:24.282435 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b88a3c99-de25-4613-bd47-29c5e503fa92 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '46dc393dda754d6fad0471ba329638c8'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:24.298078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1093/2167] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:24 2022] DELETE /identity/v3/users/46dc393dda754d6fad0471ba329638c8 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:24.302746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca7d04a-2980-4b0b-93d8-3568762352be None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:24.302974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca7d04a-2980-4b0b-93d8-3568762352be None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:24.303263 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca7d04a-2980-4b0b-93d8-3568762352be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:24.325341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-aca7d04a-2980-4b0b-93d8-3568762352be None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:24.326878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-aca7d04a-2980-4b0b-93d8-3568762352be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:24.430588 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1075/2168] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:27:24.442321 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88a3c99-de25-4613-bd47-29c5e503fa92 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:24.445953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b88a3c99-de25-4613-bd47-29c5e503fa92 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:24.553558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88a3c99-de25-4613-bd47-29c5e503fa92 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:24.557873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d79f335-7b43-49b7-9840-03edb819368f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:24.558669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d79f335-7b43-49b7-9840-03edb819368f service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:24.558969 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d79f335-7b43-49b7-9840-03edb819368f service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:24.559313 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d79f335-7b43-49b7-9840-03edb819368f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:24.572148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d79f335-7b43-49b7-9840-03edb819368f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:24.573810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d79f335-7b43-49b7-9840-03edb819368f service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:24.584608 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1094/2169] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:24.988634 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca7d04a-2980-4b0b-93d8-3568762352be None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:24.990997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-aca7d04a-2980-4b0b-93d8-3568762352be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:25.013961 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca7d04a-2980-4b0b-93d8-3568762352be None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:25.030036 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff68363e-e9a8-4fdd-88a3-e620d2df1a10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:25.030699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff68363e-e9a8-4fdd-88a3-e620d2df1a10 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:25.030998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff68363e-e9a8-4fdd-88a3-e620d2df1a10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:25.031370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff68363e-e9a8-4fdd-88a3-e620d2df1a10 admin admin] PATH_INFO: `/v3/projects/b3c661e1e7d1467884fbc66821050373` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:25.033989 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff68363e-e9a8-4fdd-88a3-e620d2df1a10 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b3c661e1e7d1467884fbc66821050373)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:25.036581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff68363e-e9a8-4fdd-88a3-e620d2df1a10 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:25.124731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d79f335-7b43-49b7-9840-03edb819368f service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:25.125460 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ff68363e-e9a8-4fdd-88a3-e620d2df1a10 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b3c661e1e7d1467884fbc66821050373', 'request_id': 'req-ff68363e-e9a8-4fdd-88a3-e620d2df1a10'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:25.131622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-7d79f335-7b43-49b7-9840-03edb819368f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:25.142763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ff68363e-e9a8-4fdd-88a3-e620d2df1a10 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b3c661e1e7d1467884fbc66821050373', 'request_id': 'req-ff68363e-e9a8-4fdd-88a3-e620d2df1a10'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:25.156062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ff68363e-e9a8-4fdd-88a3-e620d2df1a10 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b3c661e1e7d1467884fbc66821050373', 'request_id': 'req-ff68363e-e9a8-4fdd-88a3-e620d2df1a10'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:25.176609 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ff68363e-e9a8-4fdd-88a3-e620d2df1a10 admin admin] The token cache is being invalidate because project b3c661e1e7d1467884fbc66821050373 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:25.176609 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ff68363e-e9a8-4fdd-88a3-e620d2df1a10 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:25.177359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ff68363e-e9a8-4fdd-88a3-e620d2df1a10 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:25.179643 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1076/2170] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:24 2022] DELETE /identity/v3/projects/b3c661e1e7d1467884fbc66821050373 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:25.183764 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff68363e-e9a8-4fdd-88a3-e620d2df1a10 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:25.185748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ff68363e-e9a8-4fdd-88a3-e620d2df1a10 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:25.232969 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d79f335-7b43-49b7-9840-03edb819368f service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:25.262732 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff68363e-e9a8-4fdd-88a3-e620d2df1a10 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:25.268608 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b89d6d-3ceb-41dc-8423-f2c2e2ba8f39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:25.269284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b89d6d-3ceb-41dc-8423-f2c2e2ba8f39 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:25.269563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b89d6d-3ceb-41dc-8423-f2c2e2ba8f39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:25.269833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b89d6d-3ceb-41dc-8423-f2c2e2ba8f39 admin admin] PATH_INFO: `/v3/users/85b0e654512043708b06ebefe55a3fb6` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:25.271011 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b89d6d-3ceb-41dc-8423-f2c2e2ba8f39 admin admin] RBAC: Authorizing `identity:delete_user(user_id=85b0e654512043708b06ebefe55a3fb6)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:25.273010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b89d6d-3ceb-41dc-8423-f2c2e2ba8f39 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:25.279755 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:25.284508 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e70e60-e260-4c5e-ad09-4fe7dc1719f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:25.285157 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e70e60-e260-4c5e-ad09-4fe7dc1719f6 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:25.286077 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e70e60-e260-4c5e-ad09-4fe7dc1719f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:25.286354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e70e60-e260-4c5e-ad09-4fe7dc1719f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:25.288111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d8e70e60-e260-4c5e-ad09-4fe7dc1719f6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:25.325923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:DELETE=4,keystone:SELECT=95,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:25.331708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f8b89d6d-3ceb-41dc-8423-f2c2e2ba8f39 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '85b0e654512043708b06ebefe55a3fb6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:25.346209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f8b89d6d-3ceb-41dc-8423-f2c2e2ba8f39 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '85b0e654512043708b06ebefe55a3fb6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:25.353594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f8b89d6d-3ceb-41dc-8423-f2c2e2ba8f39 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '85b0e654512043708b06ebefe55a3fb6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:25.359348 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e70e60-e260-4c5e-ad09-4fe7dc1719f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:25.360893 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e70e60-e260-4c5e-ad09-4fe7dc1719f6 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:25.361450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f8b89d6d-3ceb-41dc-8423-f2c2e2ba8f39 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '85b0e654512043708b06ebefe55a3fb6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:25.362107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f8b89d6d-3ceb-41dc-8423-f2c2e2ba8f39 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '85b0e654512043708b06ebefe55a3fb6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:25.371397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1095/2171] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:25 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:25.373687 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1077/2172] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:25 2022] DELETE /identity/v3/users/85b0e654512043708b06ebefe55a3fb6 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:25.741078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e70e60-e260-4c5e-ad09-4fe7dc1719f6 service cinder] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:25.758478 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e70e60-e260-4c5e-ad09-4fe7dc1719f6 service cinder] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:25.820473 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56a8bb4-6055-4779-bdb4-2501bbd00c6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:25.821144 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56a8bb4-6055-4779-bdb4-2501bbd00c6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:25.821422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56a8bb4-6055-4779-bdb4-2501bbd00c6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:25.821690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56a8bb4-6055-4779-bdb4-2501bbd00c6b admin admin] PATH_INFO: `/v3/projects/05a904572665490e947571f5d80555d1` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:25.822817 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56a8bb4-6055-4779-bdb4-2501bbd00c6b admin admin] RBAC: Authorizing `identity:delete_project(project_id=05a904572665490e947571f5d80555d1)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:25.824776 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56a8bb4-6055-4779-bdb4-2501bbd00c6b admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:25.873009 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a56a8bb4-6055-4779-bdb4-2501bbd00c6b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '05a904572665490e947571f5d80555d1', 'request_id': 'req-a56a8bb4-6055-4779-bdb4-2501bbd00c6b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:25.879075 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a56a8bb4-6055-4779-bdb4-2501bbd00c6b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '05a904572665490e947571f5d80555d1', 'request_id': 'req-a56a8bb4-6055-4779-bdb4-2501bbd00c6b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:25.886570 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a56a8bb4-6055-4779-bdb4-2501bbd00c6b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '05a904572665490e947571f5d80555d1', 'request_id': 'req-a56a8bb4-6055-4779-bdb4-2501bbd00c6b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:25.898910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a56a8bb4-6055-4779-bdb4-2501bbd00c6b admin admin] The token cache is being invalidate because project 05a904572665490e947571f5d80555d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:25.899244 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a56a8bb4-6055-4779-bdb4-2501bbd00c6b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:25.899888 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a56a8bb4-6055-4779-bdb4-2501bbd00c6b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:25.901414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1096/2173] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:25 2022] DELETE /identity/v3/projects/05a904572665490e947571f5d80555d1 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:25.906301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8355308-3a13-4c20-ae52-524e8220aa36 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:25.906589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8355308-3a13-4c20-ae52-524e8220aa36 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:25.906887 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8355308-3a13-4c20-ae52-524e8220aa36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:25.926758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-a8355308-3a13-4c20-ae52-524e8220aa36 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:25.927973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a8355308-3a13-4c20-ae52-524e8220aa36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:25.988884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1078/2174] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:25.998027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a4bde2-a3a0-4be4-9538-227756cf1993 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:25.998310 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a4bde2-a3a0-4be4-9538-227756cf1993 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:25.998577 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a4bde2-a3a0-4be4-9538-227756cf1993 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:26.019325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-84a4bde2-a3a0-4be4-9538-227756cf1993 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:26.020620 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-84a4bde2-a3a0-4be4-9538-227756cf1993 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:26.028582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1097/2175] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:26.032854 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8355308-3a13-4c20-ae52-524e8220aa36 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:26.042823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8355308-3a13-4c20-ae52-524e8220aa36 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:26.047805 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657deeba-f8fe-4147-aa11-bd39dd12fbd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:26.048921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657deeba-f8fe-4147-aa11-bd39dd12fbd3 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:26.049237 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657deeba-f8fe-4147-aa11-bd39dd12fbd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:26.049513 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657deeba-f8fe-4147-aa11-bd39dd12fbd3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:26.050176 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657deeba-f8fe-4147-aa11-bd39dd12fbd3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:27:26.050481 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657deeba-f8fe-4147-aa11-bd39dd12fbd3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:26.052089 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657deeba-f8fe-4147-aa11-bd39dd12fbd3 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:26.065011 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1079/2176] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:27:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:27:26.069490 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a4bde2-a3a0-4be4-9538-227756cf1993 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:26.079164 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a4bde2-a3a0-4be4-9538-227756cf1993 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:26.083908 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed4ad6e-20b7-4889-bdbc-dc1a467a9623 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:26.084509 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed4ad6e-20b7-4889-bdbc-dc1a467a9623 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:26.084790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed4ad6e-20b7-4889-bdbc-dc1a467a9623 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:26.085092 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed4ad6e-20b7-4889-bdbc-dc1a467a9623 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:26.085782 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed4ad6e-20b7-4889-bdbc-dc1a467a9623 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:26.087657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed4ad6e-20b7-4889-bdbc-dc1a467a9623 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:26.122785 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1098/2177] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:26 2022] POST /identity/v3/projects => generated 409 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:27:26.124115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657deeba-f8fe-4147-aa11-bd39dd12fbd3 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:26.136015 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657deeba-f8fe-4147-aa11-bd39dd12fbd3 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:26.196954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ea7bb6-1160-41a1-b545-2e6d5df666d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:26.197580 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ea7bb6-1160-41a1-b545-2e6d5df666d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:26.197859 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ea7bb6-1160-41a1-b545-2e6d5df666d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:26.198126 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ea7bb6-1160-41a1-b545-2e6d5df666d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:26.198740 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ea7bb6-1160-41a1-b545-2e6d5df666d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:26.200736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ea7bb6-1160-41a1-b545-2e6d5df666d4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:26.225787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1080/2178] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:26 2022] POST /identity/v3/users => generated 305 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:26.226973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed4ad6e-20b7-4889-bdbc-dc1a467a9623 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:26.238080 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed4ad6e-20b7-4889-bdbc-dc1a467a9623 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:26.243130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9594ad-1390-43bc-9505-c5607363b748 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:26.244066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9594ad-1390-43bc-9505-c5607363b748 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:26.244409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9594ad-1390-43bc-9505-c5607363b748 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:26.244863 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9594ad-1390-43bc-9505-c5607363b748 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:26.245538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9594ad-1390-43bc-9505-c5607363b748 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:26.247136 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9594ad-1390-43bc-9505-c5607363b748 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:26.270321 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1099/2179] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:26 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:26.271751 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ea7bb6-1160-41a1-b545-2e6d5df666d4 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:26.277083 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:26.283916 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ea7bb6-1160-41a1-b545-2e6d5df666d4 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:26.294108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0b368a-3c43-424e-90ce-1313e7a34729 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:26.294746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0b368a-3c43-424e-90ce-1313e7a34729 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:26.295027 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0b368a-3c43-424e-90ce-1313e7a34729 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:26.295986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0b368a-3c43-424e-90ce-1313e7a34729 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:26.296602 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0b368a-3c43-424e-90ce-1313e7a34729 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:26.298970 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0b368a-3c43-424e-90ce-1313e7a34729 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:26.313633 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1081/2180] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:26 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:26.319351 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9594ad-1390-43bc-9505-c5607363b748 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:26.337107 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9594ad-1390-43bc-9505-c5607363b748 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:26.340610 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a88b742-5fea-4f97-84d4-c866342851e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:26.341306 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a88b742-5fea-4f97-84d4-c866342851e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:26.341598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a88b742-5fea-4f97-84d4-c866342851e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:26.341861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a88b742-5fea-4f97-84d4-c866342851e4 admin admin] PATH_INFO: `/v3/projects/76fcbd0cabe54191b8044788dedd3c10/users/c1a397633457459787d0884d4adfb1a6/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:26.358668 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a88b742-5fea-4f97-84d4-c866342851e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=76fcbd0cabe54191b8044788dedd3c10, user_id=c1a397633457459787d0884d4adfb1a6, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:26.362090 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a88b742-5fea-4f97-84d4-c866342851e4 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:26.366275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:26.379245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1100/2181] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:26 2022] PUT /identity/v3/projects/76fcbd0cabe54191b8044788dedd3c10/users/c1a397633457459787d0884d4adfb1a6/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:26.390614 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76420188-45ef-467d-9bef-84f503afbedd None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:26.390895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76420188-45ef-467d-9bef-84f503afbedd None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:26.391170 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76420188-45ef-467d-9bef-84f503afbedd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:26.422478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-76420188-45ef-467d-9bef-84f503afbedd None None] MFA Rules not processed for user `c1a397633457459787d0884d4adfb1a6`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:26.424322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-76420188-45ef-467d-9bef-84f503afbedd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:26.494300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1082/2182] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:26 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:26.498928 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a88b742-5fea-4f97-84d4-c866342851e4 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:26.591399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a88b742-5fea-4f97-84d4-c866342851e4 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:26.597936 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c898c0ed-431d-4d90-91d4-dc160b2c0716 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:26.598796 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c898c0ed-431d-4d90-91d4-dc160b2c0716 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:26.599181 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c898c0ed-431d-4d90-91d4-dc160b2c0716 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:26.599544 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c898c0ed-431d-4d90-91d4-dc160b2c0716 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:26.600477 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c898c0ed-431d-4d90-91d4-dc160b2c0716 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:26.602964 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c898c0ed-431d-4d90-91d4-dc160b2c0716 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:26.639756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1101/2183] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:26 2022] POST /identity/v3/projects => generated 409 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:27:26.644324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76420188-45ef-467d-9bef-84f503afbedd None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:26.658175 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76420188-45ef-467d-9bef-84f503afbedd None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:26.737536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa12225c-c90f-45ce-a79e-95135e728e97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:26.738161 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa12225c-c90f-45ce-a79e-95135e728e97 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:26.738447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa12225c-c90f-45ce-a79e-95135e728e97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:26.738820 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa12225c-c90f-45ce-a79e-95135e728e97 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:26.739348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa12225c-c90f-45ce-a79e-95135e728e97 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:26.741301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa12225c-c90f-45ce-a79e-95135e728e97 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:26.775643 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1083/2184] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:26 2022] POST /identity/v3/users => generated 311 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:27:26.777323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c898c0ed-431d-4d90-91d4-dc160b2c0716 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:26.801141 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c898c0ed-431d-4d90-91d4-dc160b2c0716 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:26.801141 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c16b5d6-e5a0-4545-b5bf-8156031fc1f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:26.801141 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c16b5d6-e5a0-4545-b5bf-8156031fc1f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:26.801141 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c16b5d6-e5a0-4545-b5bf-8156031fc1f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:26.801578 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c16b5d6-e5a0-4545-b5bf-8156031fc1f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:26.801578 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c16b5d6-e5a0-4545-b5bf-8156031fc1f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:26.801578 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c16b5d6-e5a0-4545-b5bf-8156031fc1f5 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:26.814402 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1102/2185] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:26 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:26.815730 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa12225c-c90f-45ce-a79e-95135e728e97 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:26.827083 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa12225c-c90f-45ce-a79e-95135e728e97 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:26.839592 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5046087a-3945-42a6-a53f-cc06763af9d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:26.840328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5046087a-3945-42a6-a53f-cc06763af9d0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:26.840621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5046087a-3945-42a6-a53f-cc06763af9d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:26.840868 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5046087a-3945-42a6-a53f-cc06763af9d0 admin admin] PATH_INFO: `/v3/projects/95ac22ebfad240ad9cad0f8e5b60a27a/users/db77d9d078134c189762cd00e33676f5/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:26.861241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5046087a-3945-42a6-a53f-cc06763af9d0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95ac22ebfad240ad9cad0f8e5b60a27a, user_id=db77d9d078134c189762cd00e33676f5, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:26.864198 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5046087a-3945-42a6-a53f-cc06763af9d0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:26.883933 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1084/2186] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:26 2022] PUT /identity/v3/projects/95ac22ebfad240ad9cad0f8e5b60a27a/users/db77d9d078134c189762cd00e33676f5/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:26.887614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c16b5d6-e5a0-4545-b5bf-8156031fc1f5 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:26.898195 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c16b5d6-e5a0-4545-b5bf-8156031fc1f5 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:26.963160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e8dfc5-5054-470a-9a54-b8cf68ae803c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:26.963867 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e8dfc5-5054-470a-9a54-b8cf68ae803c admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:26.964217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e8dfc5-5054-470a-9a54-b8cf68ae803c admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:26.964581 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e8dfc5-5054-470a-9a54-b8cf68ae803c admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:26.965362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e8dfc5-5054-470a-9a54-b8cf68ae803c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:26.967121 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e8dfc5-5054-470a-9a54-b8cf68ae803c admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:26.979042 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1103/2187] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:26 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:26.984375 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5046087a-3945-42a6-a53f-cc06763af9d0 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:26.995694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5046087a-3945-42a6-a53f-cc06763af9d0 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:27.000343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550abf89-f28f-4aee-b76e-a05ff95fd133 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:27.000986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550abf89-f28f-4aee-b76e-a05ff95fd133 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:27.001322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550abf89-f28f-4aee-b76e-a05ff95fd133 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:27.001592 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550abf89-f28f-4aee-b76e-a05ff95fd133 admin admin] PATH_INFO: `/v3/projects/95ac22ebfad240ad9cad0f8e5b60a27a/users/db77d9d078134c189762cd00e33676f5/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:27.003454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550abf89-f28f-4aee-b76e-a05ff95fd133 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95ac22ebfad240ad9cad0f8e5b60a27a, user_id=db77d9d078134c189762cd00e33676f5, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:27.006180 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550abf89-f28f-4aee-b76e-a05ff95fd133 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:27.016568 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1085/2188] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:26 2022] PUT /identity/v3/projects/95ac22ebfad240ad9cad0f8e5b60a27a/users/db77d9d078134c189762cd00e33676f5/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:27.028139 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d039bd95-8a12-4642-b2b5-2ddd6b457ed5 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:27.028426 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d039bd95-8a12-4642-b2b5-2ddd6b457ed5 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:27.028704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d039bd95-8a12-4642-b2b5-2ddd6b457ed5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:27.061088 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-d039bd95-8a12-4642-b2b5-2ddd6b457ed5 None None] MFA Rules not processed for user `db77d9d078134c189762cd00e33676f5`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:27.062297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d039bd95-8a12-4642-b2b5-2ddd6b457ed5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:27.122867 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:27.143092 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1104/2189] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:27 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:27.154274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550abf89-f28f-4aee-b76e-a05ff95fd133 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:27.156077 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-550abf89-f28f-4aee-b76e-a05ff95fd133 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:27.193941 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:27.220004 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550abf89-f28f-4aee-b76e-a05ff95fd133 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:27.223660 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85985d1-6223-4360-b5a7-a19b1ace4d0f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:27.224268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85985d1-6223-4360-b5a7-a19b1ace4d0f service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:27.224548 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85985d1-6223-4360-b5a7-a19b1ace4d0f service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:27.224819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85985d1-6223-4360-b5a7-a19b1ace4d0f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:27.236294 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85985d1-6223-4360-b5a7-a19b1ace4d0f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:27.237883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85985d1-6223-4360-b5a7-a19b1ace4d0f service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:27.249446 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1086/2190] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:27 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:27.465643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d039bd95-8a12-4642-b2b5-2ddd6b457ed5 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:27.483642 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d039bd95-8a12-4642-b2b5-2ddd6b457ed5 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:27.489762 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13d08a6-d752-4c14-aae4-67ec49801296 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:27.490863 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13d08a6-d752-4c14-aae4-67ec49801296 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:27.491365 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13d08a6-d752-4c14-aae4-67ec49801296 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:27.491856 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13d08a6-d752-4c14-aae4-67ec49801296 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:27.592849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13d08a6-d752-4c14-aae4-67ec49801296 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:27.594429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13d08a6-d752-4c14-aae4-67ec49801296 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:27.605975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1105/2191] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:27 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:27.774549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85985d1-6223-4360-b5a7-a19b1ace4d0f service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:27.776421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e85985d1-6223-4360-b5a7-a19b1ace4d0f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:27.893638 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85985d1-6223-4360-b5a7-a19b1ace4d0f service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:27.908247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6967108-aea7-4194-a83d-952c41fcc39f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:27.908541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6967108-aea7-4194-a83d-952c41fcc39f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:27.908823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6967108-aea7-4194-a83d-952c41fcc39f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:27.909278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6967108-aea7-4194-a83d-952c41fcc39f admin admin] PATH_INFO: `/v3/users/80a36eeae4914e5ca58c5ce538226896` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:27.910484 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6967108-aea7-4194-a83d-952c41fcc39f admin admin] RBAC: Authorizing `identity:delete_user(user_id=80a36eeae4914e5ca58c5ce538226896)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:27.912434 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6967108-aea7-4194-a83d-952c41fcc39f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:27.979483 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d6967108-aea7-4194-a83d-952c41fcc39f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80a36eeae4914e5ca58c5ce538226896'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:27.998993 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d6967108-aea7-4194-a83d-952c41fcc39f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80a36eeae4914e5ca58c5ce538226896'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:28.010122 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d6967108-aea7-4194-a83d-952c41fcc39f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80a36eeae4914e5ca58c5ce538226896'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:28.025950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d6967108-aea7-4194-a83d-952c41fcc39f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80a36eeae4914e5ca58c5ce538226896'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:28.026815 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d6967108-aea7-4194-a83d-952c41fcc39f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80a36eeae4914e5ca58c5ce538226896'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:28.042943 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1087/2192] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:27 2022] DELETE /identity/v3/users/80a36eeae4914e5ca58c5ce538226896 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:28.047291 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34b6888-874e-4c47-92b2-4b6eef06955a None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:28.047700 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34b6888-874e-4c47-92b2-4b6eef06955a None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:28.047981 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34b6888-874e-4c47-92b2-4b6eef06955a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:28.069957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-d34b6888-874e-4c47-92b2-4b6eef06955a None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:28.071700 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d34b6888-874e-4c47-92b2-4b6eef06955a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:28.166507 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1106/2193] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:28.175533 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6967108-aea7-4194-a83d-952c41fcc39f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:28.177092 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d6967108-aea7-4194-a83d-952c41fcc39f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:28.276166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6967108-aea7-4194-a83d-952c41fcc39f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:28.280058 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9653ee-30c4-45c2-9955-85f6c8847cb2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:28.280807 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9653ee-30c4-45c2-9955-85f6c8847cb2 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:28.281226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9653ee-30c4-45c2-9955-85f6c8847cb2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:28.281599 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9653ee-30c4-45c2-9955-85f6c8847cb2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:28.294211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9653ee-30c4-45c2-9955-85f6c8847cb2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:28.295681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9653ee-30c4-45c2-9955-85f6c8847cb2 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:28.300036 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:28.307181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1088/2194] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:28.712345 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34b6888-874e-4c47-92b2-4b6eef06955a None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:28.713922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d34b6888-874e-4c47-92b2-4b6eef06955a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:28.727724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34b6888-874e-4c47-92b2-4b6eef06955a None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:28.732741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c968807c-90cc-41aa-9631-65ae56439aa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:28.733450 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c968807c-90cc-41aa-9631-65ae56439aa0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:28.733797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c968807c-90cc-41aa-9631-65ae56439aa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:28.734138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c968807c-90cc-41aa-9631-65ae56439aa0 admin admin] PATH_INFO: `/v3/projects/a58ceed104724b9193d4a2f6f1ff822b` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:28.735342 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c968807c-90cc-41aa-9631-65ae56439aa0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a58ceed104724b9193d4a2f6f1ff822b)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:28.737948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c968807c-90cc-41aa-9631-65ae56439aa0 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:28.787619 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c968807c-90cc-41aa-9631-65ae56439aa0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a58ceed104724b9193d4a2f6f1ff822b', 'request_id': 'req-c968807c-90cc-41aa-9631-65ae56439aa0'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:28.793078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c968807c-90cc-41aa-9631-65ae56439aa0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a58ceed104724b9193d4a2f6f1ff822b', 'request_id': 'req-c968807c-90cc-41aa-9631-65ae56439aa0'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:28.801861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c968807c-90cc-41aa-9631-65ae56439aa0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a58ceed104724b9193d4a2f6f1ff822b', 'request_id': 'req-c968807c-90cc-41aa-9631-65ae56439aa0'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:28.813114 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c968807c-90cc-41aa-9631-65ae56439aa0 admin admin] The token cache is being invalidate because project a58ceed104724b9193d4a2f6f1ff822b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:28.813409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c968807c-90cc-41aa-9631-65ae56439aa0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:28.814592 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c968807c-90cc-41aa-9631-65ae56439aa0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:28.816163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1107/2195] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:28 2022] DELETE /identity/v3/projects/a58ceed104724b9193d4a2f6f1ff822b => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:28.820173 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9653ee-30c4-45c2-9955-85f6c8847cb2 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:28.830549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9653ee-30c4-45c2-9955-85f6c8847cb2 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:28.888082 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff93167c-78c9-47b2-b7b0-7848f9d33d90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:28.888725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff93167c-78c9-47b2-b7b0-7848f9d33d90 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:28.889007 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff93167c-78c9-47b2-b7b0-7848f9d33d90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:28.889316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff93167c-78c9-47b2-b7b0-7848f9d33d90 admin admin] PATH_INFO: `/v3/users/d2149700938d4df29d652d01029168b6` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:28.890438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff93167c-78c9-47b2-b7b0-7848f9d33d90 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d2149700938d4df29d652d01029168b6)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:28.893071 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff93167c-78c9-47b2-b7b0-7848f9d33d90 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:28.940037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ff93167c-78c9-47b2-b7b0-7848f9d33d90 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd2149700938d4df29d652d01029168b6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:28.955515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ff93167c-78c9-47b2-b7b0-7848f9d33d90 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd2149700938d4df29d652d01029168b6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:28.963025 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ff93167c-78c9-47b2-b7b0-7848f9d33d90 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd2149700938d4df29d652d01029168b6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:28.970986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ff93167c-78c9-47b2-b7b0-7848f9d33d90 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd2149700938d4df29d652d01029168b6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:28.971589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ff93167c-78c9-47b2-b7b0-7848f9d33d90 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd2149700938d4df29d652d01029168b6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:28.977696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1089/2196] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:28 2022] DELETE /identity/v3/users/d2149700938d4df29d652d01029168b6 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:29.347352 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c968807c-90cc-41aa-9631-65ae56439aa0 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:29.348988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c968807c-90cc-41aa-9631-65ae56439aa0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:29.375442 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c968807c-90cc-41aa-9631-65ae56439aa0 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:29.409352 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:29.528263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56859ec-4765-4bd1-b5ba-edda626c24a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:29.528904 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56859ec-4765-4bd1-b5ba-edda626c24a4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:29.529334 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56859ec-4765-4bd1-b5ba-edda626c24a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:29.529613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56859ec-4765-4bd1-b5ba-edda626c24a4 admin admin] PATH_INFO: `/v3/projects/d3c6aa7d022a4c919cdf4b20184646b3` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:29.530757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56859ec-4765-4bd1-b5ba-edda626c24a4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d3c6aa7d022a4c919cdf4b20184646b3)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:29.533466 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56859ec-4765-4bd1-b5ba-edda626c24a4 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:29.540160 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff93167c-78c9-47b2-b7b0-7848f9d33d90 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:29.545462 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ff93167c-78c9-47b2-b7b0-7848f9d33d90 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:29.563203 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff93167c-78c9-47b2-b7b0-7848f9d33d90 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:29.574002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579e147f-b080-421e-9fd7-1091ad4089e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:29.574663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579e147f-b080-421e-9fd7-1091ad4089e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:29.574939 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579e147f-b080-421e-9fd7-1091ad4089e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:29.575217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579e147f-b080-421e-9fd7-1091ad4089e8 admin admin] PATH_INFO: `/v3/users/789be65a6c804a24a17ad2d465754f4e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:29.577310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-579e147f-b080-421e-9fd7-1091ad4089e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=789be65a6c804a24a17ad2d465754f4e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:29.579323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-579e147f-b080-421e-9fd7-1091ad4089e8 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:29.600675 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e56859ec-4765-4bd1-b5ba-edda626c24a4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd3c6aa7d022a4c919cdf4b20184646b3', 'request_id': 'req-e56859ec-4765-4bd1-b5ba-edda626c24a4'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:29.606612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e56859ec-4765-4bd1-b5ba-edda626c24a4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd3c6aa7d022a4c919cdf4b20184646b3', 'request_id': 'req-e56859ec-4765-4bd1-b5ba-edda626c24a4'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:29.615098 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e56859ec-4765-4bd1-b5ba-edda626c24a4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd3c6aa7d022a4c919cdf4b20184646b3', 'request_id': 'req-e56859ec-4765-4bd1-b5ba-edda626c24a4'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:29.629558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e56859ec-4765-4bd1-b5ba-edda626c24a4 admin admin] The token cache is being invalidate because project d3c6aa7d022a4c919cdf4b20184646b3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:29.629850 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e56859ec-4765-4bd1-b5ba-edda626c24a4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:29.630514 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e56859ec-4765-4bd1-b5ba-edda626c24a4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:29.632274 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1108/2197] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:29 2022] DELETE /identity/v3/projects/d3c6aa7d022a4c919cdf4b20184646b3 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:29.634498 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-579e147f-b080-421e-9fd7-1091ad4089e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '789be65a6c804a24a17ad2d465754f4e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:29.637225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56859ec-4765-4bd1-b5ba-edda626c24a4 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:29.638586 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e56859ec-4765-4bd1-b5ba-edda626c24a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:29.649246 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-579e147f-b080-421e-9fd7-1091ad4089e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '789be65a6c804a24a17ad2d465754f4e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:29.655157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=80,keystone:DELETE=19,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:29.661642 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-579e147f-b080-421e-9fd7-1091ad4089e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '789be65a6c804a24a17ad2d465754f4e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:29.678384 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-579e147f-b080-421e-9fd7-1091ad4089e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '789be65a6c804a24a17ad2d465754f4e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:29.679455 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-579e147f-b080-421e-9fd7-1091ad4089e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '789be65a6c804a24a17ad2d465754f4e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:29.700410 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1090/2198] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:29 2022] DELETE /identity/v3/users/789be65a6c804a24a17ad2d465754f4e => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:29.707091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28a2b6b-90b1-45c4-9adb-27d930c543c4 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:29.707540 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28a2b6b-90b1-45c4-9adb-27d930c543c4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:29.707909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28a2b6b-90b1-45c4-9adb-27d930c543c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:29.746727 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56859ec-4765-4bd1-b5ba-edda626c24a4 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:29.747309 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-f28a2b6b-90b1-45c4-9adb-27d930c543c4 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:29.749031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f28a2b6b-90b1-45c4-9adb-27d930c543c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:29.815424 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:29.833958 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246ed705-bec0-481d-8d27-45b24c17d57b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:29.834962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246ed705-bec0-481d-8d27-45b24c17d57b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:29.835479 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246ed705-bec0-481d-8d27-45b24c17d57b admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:29.835759 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246ed705-bec0-481d-8d27-45b24c17d57b admin admin] PATH_INFO: `/v3/users/649753f2c021443abd604b4f4f8ad45a` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:29.836982 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246ed705-bec0-481d-8d27-45b24c17d57b admin admin] RBAC: Authorizing `identity:delete_user(user_id=649753f2c021443abd604b4f4f8ad45a)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:29.837843 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1091/2199] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:29.839095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246ed705-bec0-481d-8d27-45b24c17d57b admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:29.849449 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28a2b6b-90b1-45c4-9adb-27d930c543c4 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:29.851158 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f28a2b6b-90b1-45c4-9adb-27d930c543c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:29.905288 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-246ed705-bec0-481d-8d27-45b24c17d57b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '649753f2c021443abd604b4f4f8ad45a'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:29.918843 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-246ed705-bec0-481d-8d27-45b24c17d57b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '649753f2c021443abd604b4f4f8ad45a'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:29.919503 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-246ed705-bec0-481d-8d27-45b24c17d57b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '649753f2c021443abd604b4f4f8ad45a'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:29.932798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28a2b6b-90b1-45c4-9adb-27d930c543c4 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:29.937270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-246ed705-bec0-481d-8d27-45b24c17d57b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '649753f2c021443abd604b4f4f8ad45a'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:29.937995 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-246ed705-bec0-481d-8d27-45b24c17d57b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '649753f2c021443abd604b4f4f8ad45a'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:29.938133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4877c691-1286-4c97-a2d9-0d2b18318d77 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:29.938855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4877c691-1286-4c97-a2d9-0d2b18318d77 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:29.939238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4877c691-1286-4c97-a2d9-0d2b18318d77 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:29.939580 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4877c691-1286-4c97-a2d9-0d2b18318d77 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:29.941700 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4877c691-1286-4c97-a2d9-0d2b18318d77 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:29.952281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1109/2200] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:29 2022] DELETE /identity/v3/users/649753f2c021443abd604b4f4f8ad45a => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:29.955285 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4877c691-1286-4c97-a2d9-0d2b18318d77 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:29.956802 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4877c691-1286-4c97-a2d9-0d2b18318d77 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:29.971245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1092/2201] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:30.434038 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246ed705-bec0-481d-8d27-45b24c17d57b admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:30.435811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-246ed705-bec0-481d-8d27-45b24c17d57b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:30.455582 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4877c691-1286-4c97-a2d9-0d2b18318d77 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:30.457262 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4877c691-1286-4c97-a2d9-0d2b18318d77 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:30.471669 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4877c691-1286-4c97-a2d9-0d2b18318d77 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:30.529947 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246ed705-bec0-481d-8d27-45b24c17d57b admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:30.533937 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a547a2f-529c-4c5a-a58c-0ec5716e1d1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:30.534519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d494baa8-c58b-4c89-ab26-4454b460bcd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:30.534693 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a547a2f-529c-4c5a-a58c-0ec5716e1d1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:30.535105 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a547a2f-529c-4c5a-a58c-0ec5716e1d1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:30.535225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d494baa8-c58b-4c89-ab26-4454b460bcd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:30.535412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d494baa8-c58b-4c89-ab26-4454b460bcd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:30.535514 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a547a2f-529c-4c5a-a58c-0ec5716e1d1a admin admin] PATH_INFO: `/v3/projects/14d5c33e0f07495f933f33275c332822` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:30.535684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d494baa8-c58b-4c89-ab26-4454b460bcd6 admin admin] PATH_INFO: `/v3/projects/3bca1b8e1b1f49129aaa607f9afd2c33` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:30.536739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a547a2f-529c-4c5a-a58c-0ec5716e1d1a admin admin] RBAC: Authorizing `identity:delete_project(project_id=14d5c33e0f07495f933f33275c332822)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:30.536853 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d494baa8-c58b-4c89-ab26-4454b460bcd6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3bca1b8e1b1f49129aaa607f9afd2c33)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:30.538805 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a547a2f-529c-4c5a-a58c-0ec5716e1d1a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:30.538910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d494baa8-c58b-4c89-ab26-4454b460bcd6 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:30.578376 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:30.590825 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0a547a2f-529c-4c5a-a58c-0ec5716e1d1a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '14d5c33e0f07495f933f33275c332822', 'request_id': 'req-0a547a2f-529c-4c5a-a58c-0ec5716e1d1a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:30.598762 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d494baa8-c58b-4c89-ab26-4454b460bcd6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3bca1b8e1b1f49129aaa607f9afd2c33', 'request_id': 'req-d494baa8-c58b-4c89-ab26-4454b460bcd6'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:30.599757 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0a547a2f-529c-4c5a-a58c-0ec5716e1d1a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '14d5c33e0f07495f933f33275c332822', 'request_id': 'req-0a547a2f-529c-4c5a-a58c-0ec5716e1d1a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:30.605863 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d494baa8-c58b-4c89-ab26-4454b460bcd6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3bca1b8e1b1f49129aaa607f9afd2c33', 'request_id': 'req-d494baa8-c58b-4c89-ab26-4454b460bcd6'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:30.608571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0a547a2f-529c-4c5a-a58c-0ec5716e1d1a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '14d5c33e0f07495f933f33275c332822', 'request_id': 'req-0a547a2f-529c-4c5a-a58c-0ec5716e1d1a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:30.614614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d494baa8-c58b-4c89-ab26-4454b460bcd6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3bca1b8e1b1f49129aaa607f9afd2c33', 'request_id': 'req-d494baa8-c58b-4c89-ab26-4454b460bcd6'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:30.623148 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0a547a2f-529c-4c5a-a58c-0ec5716e1d1a admin admin] The token cache is being invalidate because project 14d5c33e0f07495f933f33275c332822 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:30.623564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0a547a2f-529c-4c5a-a58c-0ec5716e1d1a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:30.624298 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0a547a2f-529c-4c5a-a58c-0ec5716e1d1a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:30.624789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:30.628810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1093/2202] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:30 2022] DELETE /identity/v3/projects/14d5c33e0f07495f933f33275c332822 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:30.633304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a547a2f-529c-4c5a-a58c-0ec5716e1d1a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:30.634137 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d494baa8-c58b-4c89-ab26-4454b460bcd6 admin admin] The token cache is being invalidate because project 3bca1b8e1b1f49129aaa607f9afd2c33 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:30.634828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d494baa8-c58b-4c89-ab26-4454b460bcd6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:30.634946 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0a547a2f-529c-4c5a-a58c-0ec5716e1d1a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:30.636370 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d494baa8-c58b-4c89-ab26-4454b460bcd6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:30.642063 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1110/2203] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:30 2022] DELETE /identity/v3/projects/3bca1b8e1b1f49129aaa607f9afd2c33 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:30.649436 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc86733-3139-43ef-8be2-c3c691ae6f56 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:30.649697 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc86733-3139-43ef-8be2-c3c691ae6f56 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:30.650074 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc86733-3139-43ef-8be2-c3c691ae6f56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:30.674183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-4bc86733-3139-43ef-8be2-c3c691ae6f56 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:30.676870 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4bc86733-3139-43ef-8be2-c3c691ae6f56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:30.716567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a547a2f-529c-4c5a-a58c-0ec5716e1d1a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:30.752379 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1111/2204] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:27:30.760390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23f9fea-27aa-44c6-af17-44e92bfead22 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:30.760674 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23f9fea-27aa-44c6-af17-44e92bfead22 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:30.760951 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23f9fea-27aa-44c6-af17-44e92bfead22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:30.787391 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-d23f9fea-27aa-44c6-af17-44e92bfead22 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:30.788497 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d23f9fea-27aa-44c6-af17-44e92bfead22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:30.791091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0374a217-8d14-4a70-bc8d-860f0ebcda62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:30.791732 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0374a217-8d14-4a70-bc8d-860f0ebcda62 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:30.792015 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0374a217-8d14-4a70-bc8d-860f0ebcda62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:30.792287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0374a217-8d14-4a70-bc8d-860f0ebcda62 admin admin] PATH_INFO: `/v3/users/1779bef1781c40109927b7954da19b4a` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:30.793562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0374a217-8d14-4a70-bc8d-860f0ebcda62 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1779bef1781c40109927b7954da19b4a)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:30.795497 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0374a217-8d14-4a70-bc8d-860f0ebcda62 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:30.796622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1112/2205] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:27:30.801624 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23f9fea-27aa-44c6-af17-44e92bfead22 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:30.810714 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23f9fea-27aa-44c6-af17-44e92bfead22 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:30.815923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37296179-bd82-4098-bd45-2babbb20abce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:30.816521 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37296179-bd82-4098-bd45-2babbb20abce admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:30.816798 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37296179-bd82-4098-bd45-2babbb20abce admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:30.817085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37296179-bd82-4098-bd45-2babbb20abce admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:30.817739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37296179-bd82-4098-bd45-2babbb20abce admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:27:30.818039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37296179-bd82-4098-bd45-2babbb20abce admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:30.819641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37296179-bd82-4098-bd45-2babbb20abce admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:30.836343 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1113/2206] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:27:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:27:30.837322 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37296179-bd82-4098-bd45-2babbb20abce admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:30.846152 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0374a217-8d14-4a70-bc8d-860f0ebcda62 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1779bef1781c40109927b7954da19b4a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:30.847656 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37296179-bd82-4098-bd45-2babbb20abce admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:30.852275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea7b6d0-df68-499c-ad95-892a237515f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:30.852901 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea7b6d0-df68-499c-ad95-892a237515f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:30.853220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea7b6d0-df68-499c-ad95-892a237515f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:30.853493 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea7b6d0-df68-499c-ad95-892a237515f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:30.854145 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea7b6d0-df68-499c-ad95-892a237515f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:30.856030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea7b6d0-df68-499c-ad95-892a237515f3 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:30.858686 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0374a217-8d14-4a70-bc8d-860f0ebcda62 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1779bef1781c40109927b7954da19b4a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:30.865637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:DELETE=14,keystone:SELECT=85,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:30.868326 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0374a217-8d14-4a70-bc8d-860f0ebcda62 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1779bef1781c40109927b7954da19b4a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:30.878923 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0374a217-8d14-4a70-bc8d-860f0ebcda62 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1779bef1781c40109927b7954da19b4a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:30.879664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0374a217-8d14-4a70-bc8d-860f0ebcda62 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1779bef1781c40109927b7954da19b4a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:30.887379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1094/2207] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:30 2022] DELETE /identity/v3/users/1779bef1781c40109927b7954da19b4a => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:30.889074 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1114/2208] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:30 2022] POST /identity/v3/projects => generated 383 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:27:30.893011 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0374a217-8d14-4a70-bc8d-860f0ebcda62 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:30.894664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0374a217-8d14-4a70-bc8d-860f0ebcda62 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:30.974613 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0374a217-8d14-4a70-bc8d-860f0ebcda62 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:31.041493 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e0cc4c-2655-47da-b35a-a330faec3727 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:31.041673 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e0cc4c-2655-47da-b35a-a330faec3727 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:31.042052 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e0cc4c-2655-47da-b35a-a330faec3727 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:31.042406 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e0cc4c-2655-47da-b35a-a330faec3727 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:31.043133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e0cc4c-2655-47da-b35a-a330faec3727 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:31.045360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e0cc4c-2655-47da-b35a-a330faec3727 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:31.078213 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1095/2209] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:30 2022] POST /identity/v3/users => generated 292 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:27:31.082419 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea7b6d0-df68-499c-ad95-892a237515f3 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:31.095631 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea7b6d0-df68-499c-ad95-892a237515f3 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:31.102786 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd36c96-022e-4c34-bc8d-1cc54497fc19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:31.103399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd36c96-022e-4c34-bc8d-1cc54497fc19 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:31.103676 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd36c96-022e-4c34-bc8d-1cc54497fc19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:31.103949 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd36c96-022e-4c34-bc8d-1cc54497fc19 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:31.104591 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd36c96-022e-4c34-bc8d-1cc54497fc19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:31.106330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd36c96-022e-4c34-bc8d-1cc54497fc19 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:31.118502 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1115/2210] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:31 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:27:31.125406 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e0cc4c-2655-47da-b35a-a330faec3727 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:31.135071 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e0cc4c-2655-47da-b35a-a330faec3727 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:31.139680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5662ba-a491-44e6-a2ef-51c30e25a6b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:31.140289 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5662ba-a491-44e6-a2ef-51c30e25a6b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:31.140568 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5662ba-a491-44e6-a2ef-51c30e25a6b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:31.140842 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5662ba-a491-44e6-a2ef-51c30e25a6b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:31.141539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5662ba-a491-44e6-a2ef-51c30e25a6b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:31.143379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5662ba-a491-44e6-a2ef-51c30e25a6b9 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:31.153746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1096/2211] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:31 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:31.158462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd36c96-022e-4c34-bc8d-1cc54497fc19 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:31.168552 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd36c96-022e-4c34-bc8d-1cc54497fc19 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:31.173685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c4afa7-5f7c-4ce5-bb16-112aa617ec69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:31.174339 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c4afa7-5f7c-4ce5-bb16-112aa617ec69 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:31.174617 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c4afa7-5f7c-4ce5-bb16-112aa617ec69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:31.174921 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c4afa7-5f7c-4ce5-bb16-112aa617ec69 admin admin] PATH_INFO: `/v3/projects/cd6ef70eb8a842e6ae318a06f9883f3b/users/9584a228cade4716b04ae16cb16616c5/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:31.191673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c4afa7-5f7c-4ce5-bb16-112aa617ec69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cd6ef70eb8a842e6ae318a06f9883f3b, user_id=9584a228cade4716b04ae16cb16616c5, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:31.194578 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c4afa7-5f7c-4ce5-bb16-112aa617ec69 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:31.218418 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1116/2212] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:31 2022] PUT /identity/v3/projects/cd6ef70eb8a842e6ae318a06f9883f3b/users/9584a228cade4716b04ae16cb16616c5/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:31.218741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5662ba-a491-44e6-a2ef-51c30e25a6b9 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:31.218741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bc5662ba-a491-44e6-a2ef-51c30e25a6b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:31.224609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a33d78-4abd-42b1-a602-72e72e75d8ef None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:31.224905 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a33d78-4abd-42b1-a602-72e72e75d8ef None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:31.225231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a33d78-4abd-42b1-a602-72e72e75d8ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:31.229007 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5662ba-a491-44e6-a2ef-51c30e25a6b9 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:31.257507 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-b0a33d78-4abd-42b1-a602-72e72e75d8ef None None] MFA Rules not processed for user `9584a228cade4716b04ae16cb16616c5`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:31.258782 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b0a33d78-4abd-42b1-a602-72e72e75d8ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:31.289384 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f0be7c-4600-4eb7-bc7a-06636e89a5ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:31.290071 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f0be7c-4600-4eb7-bc7a-06636e89a5ff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:31.290355 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f0be7c-4600-4eb7-bc7a-06636e89a5ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:31.290746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f0be7c-4600-4eb7-bc7a-06636e89a5ff admin admin] PATH_INFO: `/v3/projects/9d73542d9060479280ae58f5c51d2bcb` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:31.292026 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f0be7c-4600-4eb7-bc7a-06636e89a5ff admin admin] RBAC: Authorizing `identity:delete_project(project_id=9d73542d9060479280ae58f5c51d2bcb)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:31.293949 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f0be7c-4600-4eb7-bc7a-06636e89a5ff admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:31.312091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:31.327333 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1117/2213] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:31 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:31.340536 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a33d78-4abd-42b1-a602-72e72e75d8ef None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:31.346620 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b0a33d78-4abd-42b1-a602-72e72e75d8ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:31.347185 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-52f0be7c-4600-4eb7-bc7a-06636e89a5ff admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9d73542d9060479280ae58f5c51d2bcb', 'request_id': 'req-52f0be7c-4600-4eb7-bc7a-06636e89a5ff'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:31.352824 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-52f0be7c-4600-4eb7-bc7a-06636e89a5ff admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9d73542d9060479280ae58f5c51d2bcb', 'request_id': 'req-52f0be7c-4600-4eb7-bc7a-06636e89a5ff'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:31.360669 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-52f0be7c-4600-4eb7-bc7a-06636e89a5ff admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9d73542d9060479280ae58f5c51d2bcb', 'request_id': 'req-52f0be7c-4600-4eb7-bc7a-06636e89a5ff'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:31.363378 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:31.373091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-52f0be7c-4600-4eb7-bc7a-06636e89a5ff admin admin] The token cache is being invalidate because project 9d73542d9060479280ae58f5c51d2bcb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:31.373386 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-52f0be7c-4600-4eb7-bc7a-06636e89a5ff admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:31.374051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-52f0be7c-4600-4eb7-bc7a-06636e89a5ff admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:31.375753 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1097/2214] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:31 2022] DELETE /identity/v3/projects/9d73542d9060479280ae58f5c51d2bcb => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:31.379647 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f0be7c-4600-4eb7-bc7a-06636e89a5ff admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:31.380963 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-52f0be7c-4600-4eb7-bc7a-06636e89a5ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:31.419878 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a33d78-4abd-42b1-a602-72e72e75d8ef None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:31.453875 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f0be7c-4600-4eb7-bc7a-06636e89a5ff admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:31.460015 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8dc90d-5fa6-41ec-ace0-5b8fea252f24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:31.460652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8dc90d-5fa6-41ec-ace0-5b8fea252f24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:31.460955 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8dc90d-5fa6-41ec-ace0-5b8fea252f24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:31.461280 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8dc90d-5fa6-41ec-ace0-5b8fea252f24 admin admin] PATH_INFO: `/v3/users/5981c254d4cf457cb8e3350e7ca12396` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:31.462421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8dc90d-5fa6-41ec-ace0-5b8fea252f24 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5981c254d4cf457cb8e3350e7ca12396)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:31.464369 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8dc90d-5fa6-41ec-ace0-5b8fea252f24 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:31.497248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e4e06f-e48a-48bd-b109-521a700ca850 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:31.498034 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e4e06f-e48a-48bd-b109-521a700ca850 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:31.498417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e4e06f-e48a-48bd-b109-521a700ca850 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:31.498790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e4e06f-e48a-48bd-b109-521a700ca850 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:31.500713 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-84e4e06f-e48a-48bd-b109-521a700ca850 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:31.545336 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3a8dc90d-5fa6-41ec-ace0-5b8fea252f24 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5981c254d4cf457cb8e3350e7ca12396'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:31.558689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3a8dc90d-5fa6-41ec-ace0-5b8fea252f24 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5981c254d4cf457cb8e3350e7ca12396'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:31.566519 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3a8dc90d-5fa6-41ec-ace0-5b8fea252f24 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5981c254d4cf457cb8e3350e7ca12396'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:31.575466 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3a8dc90d-5fa6-41ec-ace0-5b8fea252f24 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5981c254d4cf457cb8e3350e7ca12396'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:31.575768 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e4e06f-e48a-48bd-b109-521a700ca850 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:31.576384 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3a8dc90d-5fa6-41ec-ace0-5b8fea252f24 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5981c254d4cf457cb8e3350e7ca12396'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:31.577591 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e4e06f-e48a-48bd-b109-521a700ca850 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:31.583966 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1098/2215] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:31 2022] DELETE /identity/v3/users/5981c254d4cf457cb8e3350e7ca12396 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:31.588178 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1118/2216] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:31 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 249 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:27:31.905213 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8dc90d-5fa6-41ec-ace0-5b8fea252f24 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:31.906885 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3a8dc90d-5fa6-41ec-ace0-5b8fea252f24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:31.963531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:32.011211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8dc90d-5fa6-41ec-ace0-5b8fea252f24 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:32.015806 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa31cff5-edcf-4090-9d1f-7cf92dc62d59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:32.016437 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa31cff5-edcf-4090-9d1f-7cf92dc62d59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:32.016743 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa31cff5-edcf-4090-9d1f-7cf92dc62d59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:32.017019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa31cff5-edcf-4090-9d1f-7cf92dc62d59 admin admin] PATH_INFO: `/v3/projects/f510dd6017c44d3aaf001567f00cb76b` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:32.018164 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa31cff5-edcf-4090-9d1f-7cf92dc62d59 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f510dd6017c44d3aaf001567f00cb76b)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:32.020086 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa31cff5-edcf-4090-9d1f-7cf92dc62d59 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:32.069023 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-fa31cff5-edcf-4090-9d1f-7cf92dc62d59 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f510dd6017c44d3aaf001567f00cb76b', 'request_id': 'req-fa31cff5-edcf-4090-9d1f-7cf92dc62d59'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:32.074609 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-fa31cff5-edcf-4090-9d1f-7cf92dc62d59 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f510dd6017c44d3aaf001567f00cb76b', 'request_id': 'req-fa31cff5-edcf-4090-9d1f-7cf92dc62d59'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:32.082474 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-fa31cff5-edcf-4090-9d1f-7cf92dc62d59 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f510dd6017c44d3aaf001567f00cb76b', 'request_id': 'req-fa31cff5-edcf-4090-9d1f-7cf92dc62d59'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:32.093608 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-fa31cff5-edcf-4090-9d1f-7cf92dc62d59 admin admin] The token cache is being invalidate because project f510dd6017c44d3aaf001567f00cb76b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:32.094049 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-fa31cff5-edcf-4090-9d1f-7cf92dc62d59 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:32.094647 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-fa31cff5-edcf-4090-9d1f-7cf92dc62d59 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:32.096619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1099/2217] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:31 2022] DELETE /identity/v3/projects/f510dd6017c44d3aaf001567f00cb76b => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:32.101363 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7676fa1-d9df-490e-b865-7158813e4135 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:32.101725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7676fa1-d9df-490e-b865-7158813e4135 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:32.102048 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7676fa1-d9df-490e-b865-7158813e4135 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:32.135675 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-a7676fa1-d9df-490e-b865-7158813e4135 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:32.136928 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-a7676fa1-d9df-490e-b865-7158813e4135 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:32.201547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1119/2218] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:32.209342 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa7ff73-78c8-4395-beaa-880ca6fa9802 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:32.209677 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa7ff73-78c8-4395-beaa-880ca6fa9802 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:32.209947 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa7ff73-78c8-4395-beaa-880ca6fa9802 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:32.230499 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-baa7ff73-78c8-4395-beaa-880ca6fa9802 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:32.231705 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-baa7ff73-78c8-4395-beaa-880ca6fa9802 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:32.240248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1100/2219] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:32.244397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7676fa1-d9df-490e-b865-7158813e4135 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:32.254495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7676fa1-d9df-490e-b865-7158813e4135 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:32.259026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c09e71-74f5-484c-bfec-89108dece326 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:32.259618 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c09e71-74f5-484c-bfec-89108dece326 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:32.259896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c09e71-74f5-484c-bfec-89108dece326 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:32.260171 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c09e71-74f5-484c-bfec-89108dece326 admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:32.260816 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c09e71-74f5-484c-bfec-89108dece326 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:27:32.261156 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c09e71-74f5-484c-bfec-89108dece326 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:32.262761 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c09e71-74f5-484c-bfec-89108dece326 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:32.274128 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1120/2220] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:27:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:27:32.278321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa7ff73-78c8-4395-beaa-880ca6fa9802 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:32.287531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa7ff73-78c8-4395-beaa-880ca6fa9802 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:32.292098 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a7aafb-c228-4654-961b-67dc02ba5697 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:32.292690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a7aafb-c228-4654-961b-67dc02ba5697 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:32.292971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a7aafb-c228-4654-961b-67dc02ba5697 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:32.293281 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a7aafb-c228-4654-961b-67dc02ba5697 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:32.293921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a7aafb-c228-4654-961b-67dc02ba5697 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:32.295846 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a7aafb-c228-4654-961b-67dc02ba5697 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:32.326171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1101/2221] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:32 2022] POST /identity/v3/projects => generated 371 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:32.327356 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c09e71-74f5-484c-bfec-89108dece326 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:32.337212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c09e71-74f5-484c-bfec-89108dece326 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:32.345318 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:32.393974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d48b90e-166f-47f8-b325-e5c5b2097497 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:32.394571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d48b90e-166f-47f8-b325-e5c5b2097497 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:32.394851 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d48b90e-166f-47f8-b325-e5c5b2097497 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:32.395122 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d48b90e-166f-47f8-b325-e5c5b2097497 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:32.395740 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d48b90e-166f-47f8-b325-e5c5b2097497 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:32.397751 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d48b90e-166f-47f8-b325-e5c5b2097497 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:32.418387 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1121/2222] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:32 2022] POST /identity/v3/users => generated 286 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:32.422592 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a7aafb-c228-4654-961b-67dc02ba5697 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:32.433279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a7aafb-c228-4654-961b-67dc02ba5697 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:32.437701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da7556a-07d1-4f18-94b2-8b926aa62561 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:32.438285 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da7556a-07d1-4f18-94b2-8b926aa62561 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:32.438560 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da7556a-07d1-4f18-94b2-8b926aa62561 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:32.438831 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da7556a-07d1-4f18-94b2-8b926aa62561 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:32.439469 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da7556a-07d1-4f18-94b2-8b926aa62561 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:32.441136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da7556a-07d1-4f18-94b2-8b926aa62561 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:32.450954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1102/2223] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:32 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:32.454745 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d48b90e-166f-47f8-b325-e5c5b2097497 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:32.464880 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d48b90e-166f-47f8-b325-e5c5b2097497 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:32.470301 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f819fc3a-8699-46dd-a439-aeffbf5bef0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:32.470920 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f819fc3a-8699-46dd-a439-aeffbf5bef0b admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:32.471183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f819fc3a-8699-46dd-a439-aeffbf5bef0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:32.471456 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f819fc3a-8699-46dd-a439-aeffbf5bef0b admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:32.472103 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f819fc3a-8699-46dd-a439-aeffbf5bef0b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:32.473755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f819fc3a-8699-46dd-a439-aeffbf5bef0b admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:32.484609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1122/2224] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:32 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:32.489277 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da7556a-07d1-4f18-94b2-8b926aa62561 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:32.499406 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da7556a-07d1-4f18-94b2-8b926aa62561 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:32.503983 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7297437b-35e7-46d5-adb5-1f1e9e920bea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:32.504618 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7297437b-35e7-46d5-adb5-1f1e9e920bea admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:32.504897 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7297437b-35e7-46d5-adb5-1f1e9e920bea admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:32.505200 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7297437b-35e7-46d5-adb5-1f1e9e920bea admin admin] PATH_INFO: `/v3/projects/bdbf61cc87384ade8eb07b032bb6292b/users/148a7e11786a4a54909c4a0679407b80/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:32.519079 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7297437b-35e7-46d5-adb5-1f1e9e920bea admin admin] RBAC: Authorizing `identity:create_grant(project_id=bdbf61cc87384ade8eb07b032bb6292b, user_id=148a7e11786a4a54909c4a0679407b80, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:32.521801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7297437b-35e7-46d5-adb5-1f1e9e920bea admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:32.534666 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1103/2225] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:32 2022] PUT /identity/v3/projects/bdbf61cc87384ade8eb07b032bb6292b/users/148a7e11786a4a54909c4a0679407b80/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:32.539552 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76435f12-b005-4b26-a8e7-b0aa13a439a9 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:32.540040 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76435f12-b005-4b26-a8e7-b0aa13a439a9 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:32.540154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76435f12-b005-4b26-a8e7-b0aa13a439a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:32.559549 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-76435f12-b005-4b26-a8e7-b0aa13a439a9 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:32.560781 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-76435f12-b005-4b26-a8e7-b0aa13a439a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:32.620975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1123/2226] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:32.630817 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7297437b-35e7-46d5-adb5-1f1e9e920bea admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:32.632893 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7297437b-35e7-46d5-adb5-1f1e9e920bea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:32.691571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7297437b-35e7-46d5-adb5-1f1e9e920bea admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:32.695324 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0075f61-285d-4b02-ac50-7bfc1ee868d9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:32.695936 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0075f61-285d-4b02-ac50-7bfc1ee868d9 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:32.696217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0075f61-285d-4b02-ac50-7bfc1ee868d9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:32.696490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0075f61-285d-4b02-ac50-7bfc1ee868d9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:32.702892 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:32.707912 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0075f61-285d-4b02-ac50-7bfc1ee868d9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:32.710090 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0075f61-285d-4b02-ac50-7bfc1ee868d9 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:32.723254 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1104/2227] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:33.378076 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40783d2-5ed3-4b13-8a65-f601c88e24fa None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:33.378368 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40783d2-5ed3-4b13-8a65-f601c88e24fa None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:33.378676 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40783d2-5ed3-4b13-8a65-f601c88e24fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:33.408193 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-f40783d2-5ed3-4b13-8a65-f601c88e24fa None None] MFA Rules not processed for user `148a7e11786a4a54909c4a0679407b80`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:33.409547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f40783d2-5ed3-4b13-8a65-f601c88e24fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:33.474019 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1124/2228] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:33 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:33.484603 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0075f61-285d-4b02-ac50-7bfc1ee868d9 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:33.487015 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f0075f61-285d-4b02-ac50-7bfc1ee868d9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:33.554115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0075f61-285d-4b02-ac50-7bfc1ee868d9 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:33.558436 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcecf26e-5027-4edc-952b-f95fb0e6de1c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:33.559077 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcecf26e-5027-4edc-952b-f95fb0e6de1c service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:33.559332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcecf26e-5027-4edc-952b-f95fb0e6de1c service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:33.559603 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcecf26e-5027-4edc-952b-f95fb0e6de1c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:33.569736 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcecf26e-5027-4edc-952b-f95fb0e6de1c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:33.571189 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcecf26e-5027-4edc-952b-f95fb0e6de1c service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:33.580245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1105/2229] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:33 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:34.939966 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada7df66-de9a-4629-b4d1-9d4f999cc529 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:34.940510 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada7df66-de9a-4629-b4d1-9d4f999cc529 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:34.940510 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada7df66-de9a-4629-b4d1-9d4f999cc529 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:34.964264 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-ada7df66-de9a-4629-b4d1-9d4f999cc529 None None] MFA Rules not processed for user `148a7e11786a4a54909c4a0679407b80`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:34.965518 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ada7df66-de9a-4629-b4d1-9d4f999cc529 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:34.972734 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1125/2230] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:34 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:36.298267 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcecf26e-5027-4edc-952b-f95fb0e6de1c service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:36.299717 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-dcecf26e-5027-4edc-952b-f95fb0e6de1c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:36.314271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcecf26e-5027-4edc-952b-f95fb0e6de1c service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:36.318772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80bef3a-288e-47a9-8356-2c54abde9c00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:36.319391 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80bef3a-288e-47a9-8356-2c54abde9c00 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:36.319671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80bef3a-288e-47a9-8356-2c54abde9c00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:36.319943 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80bef3a-288e-47a9-8356-2c54abde9c00 admin admin] PATH_INFO: `/v3/users/9584a228cade4716b04ae16cb16616c5` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:36.321140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80bef3a-288e-47a9-8356-2c54abde9c00 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9584a228cade4716b04ae16cb16616c5)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:36.323065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80bef3a-288e-47a9-8356-2c54abde9c00 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:36.372559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a80bef3a-288e-47a9-8356-2c54abde9c00 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9584a228cade4716b04ae16cb16616c5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:36.384853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a80bef3a-288e-47a9-8356-2c54abde9c00 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9584a228cade4716b04ae16cb16616c5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:36.391827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a80bef3a-288e-47a9-8356-2c54abde9c00 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9584a228cade4716b04ae16cb16616c5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:36.399171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a80bef3a-288e-47a9-8356-2c54abde9c00 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9584a228cade4716b04ae16cb16616c5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:36.399791 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a80bef3a-288e-47a9-8356-2c54abde9c00 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9584a228cade4716b04ae16cb16616c5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:36.405644 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1106/2231] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:36 2022] DELETE /identity/v3/users/9584a228cade4716b04ae16cb16616c5 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:36.410680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ff3c78-e73d-41ab-b7b7-1a5c3b4c2980 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:36.411179 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ff3c78-e73d-41ab-b7b7-1a5c3b4c2980 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:36.411270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ff3c78-e73d-41ab-b7b7-1a5c3b4c2980 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:36.430613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-f3ff3c78-e73d-41ab-b7b7-1a5c3b4c2980 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:36.431876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f3ff3c78-e73d-41ab-b7b7-1a5c3b4c2980 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:36.440118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:36.495965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1126/2232] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:36.505926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80bef3a-288e-47a9-8356-2c54abde9c00 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:36.507290 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a80bef3a-288e-47a9-8356-2c54abde9c00 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:36.568120 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80bef3a-288e-47a9-8356-2c54abde9c00 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:36.571997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cf4e13-88b2-4993-b0cf-7f019cde84f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:36.572703 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cf4e13-88b2-4993-b0cf-7f019cde84f8 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:36.573095 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cf4e13-88b2-4993-b0cf-7f019cde84f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:36.573444 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cf4e13-88b2-4993-b0cf-7f019cde84f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:36.583176 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04cf4e13-88b2-4993-b0cf-7f019cde84f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:36.584684 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04cf4e13-88b2-4993-b0cf-7f019cde84f8 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:36.594276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1107/2233] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:36.921897 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ff3c78-e73d-41ab-b7b7-1a5c3b4c2980 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:36.924221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f3ff3c78-e73d-41ab-b7b7-1a5c3b4c2980 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:36.948109 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ff3c78-e73d-41ab-b7b7-1a5c3b4c2980 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:36.955321 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5729fe98-fda8-4cec-b1b3-aca4f46aaac3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:36.956217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5729fe98-fda8-4cec-b1b3-aca4f46aaac3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:36.956498 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5729fe98-fda8-4cec-b1b3-aca4f46aaac3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:36.956773 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5729fe98-fda8-4cec-b1b3-aca4f46aaac3 admin admin] PATH_INFO: `/v3/projects/cd6ef70eb8a842e6ae318a06f9883f3b` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:36.957962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5729fe98-fda8-4cec-b1b3-aca4f46aaac3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cd6ef70eb8a842e6ae318a06f9883f3b)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:36.959943 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5729fe98-fda8-4cec-b1b3-aca4f46aaac3 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:37.011758 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-5729fe98-fda8-4cec-b1b3-aca4f46aaac3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cd6ef70eb8a842e6ae318a06f9883f3b', 'request_id': 'req-5729fe98-fda8-4cec-b1b3-aca4f46aaac3'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:37.017978 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-5729fe98-fda8-4cec-b1b3-aca4f46aaac3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cd6ef70eb8a842e6ae318a06f9883f3b', 'request_id': 'req-5729fe98-fda8-4cec-b1b3-aca4f46aaac3'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:37.025166 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-5729fe98-fda8-4cec-b1b3-aca4f46aaac3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cd6ef70eb8a842e6ae318a06f9883f3b', 'request_id': 'req-5729fe98-fda8-4cec-b1b3-aca4f46aaac3'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:37.036347 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-5729fe98-fda8-4cec-b1b3-aca4f46aaac3 admin admin] The token cache is being invalidate because project cd6ef70eb8a842e6ae318a06f9883f3b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:37.036637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-5729fe98-fda8-4cec-b1b3-aca4f46aaac3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:37.037291 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-5729fe98-fda8-4cec-b1b3-aca4f46aaac3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:37.038802 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1127/2234] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:36 2022] DELETE /identity/v3/projects/cd6ef70eb8a842e6ae318a06f9883f3b => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:37.044244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7756a3a-d2e4-40cb-8c18-53a937f37cde None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:37.044528 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7756a3a-d2e4-40cb-8c18-53a937f37cde None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:37.044833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7756a3a-d2e4-40cb-8c18-53a937f37cde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:37.072316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b7756a3a-d2e4-40cb-8c18-53a937f37cde None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:37.073649 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b7756a3a-d2e4-40cb-8c18-53a937f37cde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:37.083348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:37.149624 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1108/2235] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:37.158224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e237ab21-9dba-4d55-be91-0ee2623c1831 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:37.158519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e237ab21-9dba-4d55-be91-0ee2623c1831 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:37.158794 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e237ab21-9dba-4d55-be91-0ee2623c1831 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:37.179635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-e237ab21-9dba-4d55-be91-0ee2623c1831 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:37.181298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e237ab21-9dba-4d55-be91-0ee2623c1831 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:37.190148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1128/2236] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:37.195913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7756a3a-d2e4-40cb-8c18-53a937f37cde None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:37.210177 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7756a3a-d2e4-40cb-8c18-53a937f37cde None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:37.215834 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a5df94-2a6f-49f5-aa14-e8feef4191ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:37.216442 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a5df94-2a6f-49f5-aa14-e8feef4191ba admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:37.216721 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a5df94-2a6f-49f5-aa14-e8feef4191ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:37.216993 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a5df94-2a6f-49f5-aa14-e8feef4191ba admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:37.217682 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a5df94-2a6f-49f5-aa14-e8feef4191ba admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:27:37.218057 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a5df94-2a6f-49f5-aa14-e8feef4191ba admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:37.222373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a5df94-2a6f-49f5-aa14-e8feef4191ba admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:37.238277 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1109/2237] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:27:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:27:37.239752 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e237ab21-9dba-4d55-be91-0ee2623c1831 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:37.249534 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e237ab21-9dba-4d55-be91-0ee2623c1831 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:37.255255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a528d754-e7f6-4e99-91e7-70fcc33b884c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:37.255850 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a528d754-e7f6-4e99-91e7-70fcc33b884c admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:37.256153 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a528d754-e7f6-4e99-91e7-70fcc33b884c admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:37.256422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a528d754-e7f6-4e99-91e7-70fcc33b884c admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:37.257125 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a528d754-e7f6-4e99-91e7-70fcc33b884c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:37.258963 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a528d754-e7f6-4e99-91e7-70fcc33b884c admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:37.285700 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1129/2238] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:37 2022] POST /identity/v3/projects => generated 369 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:37.289946 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a5df94-2a6f-49f5-aa14-e8feef4191ba admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:37.300645 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a5df94-2a6f-49f5-aa14-e8feef4191ba admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:37.357786 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85938436-3b4f-4730-a7c6-29675058907c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:37.358776 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85938436-3b4f-4730-a7c6-29675058907c admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:37.359057 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85938436-3b4f-4730-a7c6-29675058907c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:37.359331 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85938436-3b4f-4730-a7c6-29675058907c admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:37.359968 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85938436-3b4f-4730-a7c6-29675058907c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:37.362793 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85938436-3b4f-4730-a7c6-29675058907c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:37.394807 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1110/2239] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:37 2022] POST /identity/v3/users => generated 285 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:37.401704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a528d754-e7f6-4e99-91e7-70fcc33b884c admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:37.419422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a528d754-e7f6-4e99-91e7-70fcc33b884c admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:37.427961 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0414fc6-c9f8-46f2-8ace-57db97b93f06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:37.428894 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0414fc6-c9f8-46f2-8ace-57db97b93f06 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:37.429408 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0414fc6-c9f8-46f2-8ace-57db97b93f06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:37.429880 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0414fc6-c9f8-46f2-8ace-57db97b93f06 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:37.430956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0414fc6-c9f8-46f2-8ace-57db97b93f06 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:37.434147 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0414fc6-c9f8-46f2-8ace-57db97b93f06 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:37.459482 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1130/2240] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:37 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:37.466738 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85938436-3b4f-4730-a7c6-29675058907c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:37.483755 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85938436-3b4f-4730-a7c6-29675058907c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:37.497248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87c4792-7552-4f59-a786-d6743ba19548 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:37.497909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87c4792-7552-4f59-a786-d6743ba19548 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:37.498188 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87c4792-7552-4f59-a786-d6743ba19548 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:37.498462 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87c4792-7552-4f59-a786-d6743ba19548 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:37.499117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c87c4792-7552-4f59-a786-d6743ba19548 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:37.500735 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c87c4792-7552-4f59-a786-d6743ba19548 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:37.512815 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1111/2241] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:37 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:37.517015 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0414fc6-c9f8-46f2-8ace-57db97b93f06 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:37.527046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0414fc6-c9f8-46f2-8ace-57db97b93f06 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:37.531971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a0bb74-8841-44fe-aeb6-6a7dffe1053f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:37.532838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a0bb74-8841-44fe-aeb6-6a7dffe1053f admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:37.533142 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a0bb74-8841-44fe-aeb6-6a7dffe1053f admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:37.533582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a0bb74-8841-44fe-aeb6-6a7dffe1053f admin admin] PATH_INFO: `/v3/projects/74880f6720d34c2f9257465620224459/users/e99b62187e244a7e8d4144f9357f1499/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:37.549728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a0bb74-8841-44fe-aeb6-6a7dffe1053f admin admin] RBAC: Authorizing `identity:create_grant(project_id=74880f6720d34c2f9257465620224459, user_id=e99b62187e244a7e8d4144f9357f1499, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:37.552943 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a0bb74-8841-44fe-aeb6-6a7dffe1053f admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:37.567687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1131/2242] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:37 2022] PUT /identity/v3/projects/74880f6720d34c2f9257465620224459/users/e99b62187e244a7e8d4144f9357f1499/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:37.579809 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae2a2ac-6951-4b8f-b1f1-d917394ba44a None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:37.580125 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae2a2ac-6951-4b8f-b1f1-d917394ba44a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:37.580408 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae2a2ac-6951-4b8f-b1f1-d917394ba44a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:37.608408 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-2ae2a2ac-6951-4b8f-b1f1-d917394ba44a None None] MFA Rules not processed for user `e99b62187e244a7e8d4144f9357f1499`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:37.609649 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2ae2a2ac-6951-4b8f-b1f1-d917394ba44a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:37.672829 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1112/2243] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:37 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:37.684775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a0bb74-8841-44fe-aeb6-6a7dffe1053f admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:37.686507 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d0a0bb74-8841-44fe-aeb6-6a7dffe1053f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:37.720880 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:37.752319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a0bb74-8841-44fe-aeb6-6a7dffe1053f admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:37.756052 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8cd87f-2da7-4611-8bd8-62de62555a23 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '354b8228661c4369b3bca2e631f3f140', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:37.756665 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8cd87f-2da7-4611-8bd8-62de62555a23 service glance] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:37.756946 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8cd87f-2da7-4611-8bd8-62de62555a23 service glance] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:37.757258 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8cd87f-2da7-4611-8bd8-62de62555a23 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:37.766251 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8cd87f-2da7-4611-8bd8-62de62555a23 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:37.767750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8cd87f-2da7-4611-8bd8-62de62555a23 service glance] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:37.776716 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1132/2244] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:37 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:38.380034 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae2a2ac-6951-4b8f-b1f1-d917394ba44a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:38.381482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2ae2a2ac-6951-4b8f-b1f1-d917394ba44a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:38.391066 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:38.447634 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae2a2ac-6951-4b8f-b1f1-d917394ba44a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:38.452020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3def943-f061-440b-b98b-edd61336a212 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:38.452638 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3def943-f061-440b-b98b-edd61336a212 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:38.452913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3def943-f061-440b-b98b-edd61336a212 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:38.453226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3def943-f061-440b-b98b-edd61336a212 admin admin] PATH_INFO: `/v3/users/c1a397633457459787d0884d4adfb1a6` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:38.454980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3def943-f061-440b-b98b-edd61336a212 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c1a397633457459787d0884d4adfb1a6)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:38.456894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3def943-f061-440b-b98b-edd61336a212 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:38.504657 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a3def943-f061-440b-b98b-edd61336a212 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c1a397633457459787d0884d4adfb1a6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:38.517588 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a3def943-f061-440b-b98b-edd61336a212 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c1a397633457459787d0884d4adfb1a6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:38.524749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a3def943-f061-440b-b98b-edd61336a212 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c1a397633457459787d0884d4adfb1a6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:38.532254 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a3def943-f061-440b-b98b-edd61336a212 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c1a397633457459787d0884d4adfb1a6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:38.532945 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a3def943-f061-440b-b98b-edd61336a212 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c1a397633457459787d0884d4adfb1a6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:38.539088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1113/2245] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:38 2022] DELETE /identity/v3/users/c1a397633457459787d0884d4adfb1a6 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:38.543890 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c03f053-be3d-460e-84ec-9de51057301e None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:38.544335 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c03f053-be3d-460e-84ec-9de51057301e None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:38.544654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c03f053-be3d-460e-84ec-9de51057301e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:38.564737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-6c03f053-be3d-460e-84ec-9de51057301e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:38.565963 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-6c03f053-be3d-460e-84ec-9de51057301e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:38.635969 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1133/2246] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:38.647182 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3def943-f061-440b-b98b-edd61336a212 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:38.648782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a3def943-f061-440b-b98b-edd61336a212 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:38.715413 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3def943-f061-440b-b98b-edd61336a212 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:38.719069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4897ddf-4220-426a-9978-214c8982a704 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:38.719855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4897ddf-4220-426a-9978-214c8982a704 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:38.720060 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4897ddf-4220-426a-9978-214c8982a704 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:38.720328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4897ddf-4220-426a-9978-214c8982a704 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:38.730082 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4897ddf-4220-426a-9978-214c8982a704 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:38.731514 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4897ddf-4220-426a-9978-214c8982a704 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:38.741857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1114/2247] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:39.337875 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c03f053-be3d-460e-84ec-9de51057301e None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:39.339836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-6c03f053-be3d-460e-84ec-9de51057301e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:39.354608 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c03f053-be3d-460e-84ec-9de51057301e None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:39.369285 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c9d435-68b5-423e-a998-e504c7051868 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:39.369856 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c9d435-68b5-423e-a998-e504c7051868 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:39.370240 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c9d435-68b5-423e-a998-e504c7051868 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:39.370401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c9d435-68b5-423e-a998-e504c7051868 admin admin] PATH_INFO: `/v3/projects/76fcbd0cabe54191b8044788dedd3c10` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:39.371664 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c9d435-68b5-423e-a998-e504c7051868 admin admin] RBAC: Authorizing `identity:delete_project(project_id=76fcbd0cabe54191b8044788dedd3c10)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:39.373966 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c9d435-68b5-423e-a998-e504c7051868 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:39.428963 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b8c9d435-68b5-423e-a998-e504c7051868 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '76fcbd0cabe54191b8044788dedd3c10', 'request_id': 'req-b8c9d435-68b5-423e-a998-e504c7051868'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:39.436485 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b8c9d435-68b5-423e-a998-e504c7051868 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '76fcbd0cabe54191b8044788dedd3c10', 'request_id': 'req-b8c9d435-68b5-423e-a998-e504c7051868'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:39.444617 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b8c9d435-68b5-423e-a998-e504c7051868 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '76fcbd0cabe54191b8044788dedd3c10', 'request_id': 'req-b8c9d435-68b5-423e-a998-e504c7051868'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:39.458454 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b8c9d435-68b5-423e-a998-e504c7051868 admin admin] The token cache is being invalidate because project 76fcbd0cabe54191b8044788dedd3c10 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:39.458752 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b8c9d435-68b5-423e-a998-e504c7051868 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:39.459412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b8c9d435-68b5-423e-a998-e504c7051868 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:39.460986 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1134/2248] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:39 2022] DELETE /identity/v3/projects/76fcbd0cabe54191b8044788dedd3c10 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:39.464866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4897ddf-4220-426a-9978-214c8982a704 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:39.476535 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4897ddf-4220-426a-9978-214c8982a704 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:39.508459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:39.563296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918f550d-bfcb-49be-b97e-4856a7c1f2d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:39.564104 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918f550d-bfcb-49be-b97e-4856a7c1f2d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:39.564391 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918f550d-bfcb-49be-b97e-4856a7c1f2d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:39.564666 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918f550d-bfcb-49be-b97e-4856a7c1f2d4 admin admin] PATH_INFO: `/v3/users/db77d9d078134c189762cd00e33676f5` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:39.566018 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918f550d-bfcb-49be-b97e-4856a7c1f2d4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=db77d9d078134c189762cd00e33676f5)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:39.568088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918f550d-bfcb-49be-b97e-4856a7c1f2d4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:39.645792 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-918f550d-bfcb-49be-b97e-4856a7c1f2d4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'db77d9d078134c189762cd00e33676f5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:39.664741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-918f550d-bfcb-49be-b97e-4856a7c1f2d4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'db77d9d078134c189762cd00e33676f5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:39.672498 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-918f550d-bfcb-49be-b97e-4856a7c1f2d4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'db77d9d078134c189762cd00e33676f5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:39.682657 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-918f550d-bfcb-49be-b97e-4856a7c1f2d4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'db77d9d078134c189762cd00e33676f5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:39.683748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-918f550d-bfcb-49be-b97e-4856a7c1f2d4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'db77d9d078134c189762cd00e33676f5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:39.694335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1115/2249] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:39 2022] DELETE /identity/v3/users/db77d9d078134c189762cd00e33676f5 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:40.004515 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c9d435-68b5-423e-a998-e504c7051868 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:40.004515 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b8c9d435-68b5-423e-a998-e504c7051868 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:40.020382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c9d435-68b5-423e-a998-e504c7051868 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:40.103618 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14cc9153-b868-4506-a99a-0d5e3e4a9d32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:40.104274 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cc9153-b868-4506-a99a-0d5e3e4a9d32 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:40.104679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cc9153-b868-4506-a99a-0d5e3e4a9d32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:40.104954 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cc9153-b868-4506-a99a-0d5e3e4a9d32 admin admin] PATH_INFO: `/v3/projects/95ac22ebfad240ad9cad0f8e5b60a27a` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:40.106587 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14cc9153-b868-4506-a99a-0d5e3e4a9d32 admin admin] RBAC: Authorizing `identity:delete_project(project_id=95ac22ebfad240ad9cad0f8e5b60a27a)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:40.108501 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14cc9153-b868-4506-a99a-0d5e3e4a9d32 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:40.110169 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:40.169136 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-14cc9153-b868-4506-a99a-0d5e3e4a9d32 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95ac22ebfad240ad9cad0f8e5b60a27a', 'request_id': 'req-14cc9153-b868-4506-a99a-0d5e3e4a9d32'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:40.174958 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-14cc9153-b868-4506-a99a-0d5e3e4a9d32 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95ac22ebfad240ad9cad0f8e5b60a27a', 'request_id': 'req-14cc9153-b868-4506-a99a-0d5e3e4a9d32'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:40.182632 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-14cc9153-b868-4506-a99a-0d5e3e4a9d32 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95ac22ebfad240ad9cad0f8e5b60a27a', 'request_id': 'req-14cc9153-b868-4506-a99a-0d5e3e4a9d32'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:40.195412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-14cc9153-b868-4506-a99a-0d5e3e4a9d32 admin admin] The token cache is being invalidate because project 95ac22ebfad240ad9cad0f8e5b60a27a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:40.195707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-14cc9153-b868-4506-a99a-0d5e3e4a9d32 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:40.196374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-14cc9153-b868-4506-a99a-0d5e3e4a9d32 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:40.197950 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1135/2250] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:39 2022] DELETE /identity/v3/projects/95ac22ebfad240ad9cad0f8e5b60a27a => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:40.202767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70be0280-3301-47fb-8af3-b2fb29ac4971 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:40.203136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70be0280-3301-47fb-8af3-b2fb29ac4971 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:40.203475 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70be0280-3301-47fb-8af3-b2fb29ac4971 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:40.224180 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-70be0280-3301-47fb-8af3-b2fb29ac4971 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:40.225704 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-70be0280-3301-47fb-8af3-b2fb29ac4971 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:40.295317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1116/2251] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:27:40.303640 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a52a5f-ec9c-4216-aad7-5dcea933e371 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:40.304016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a52a5f-ec9c-4216-aad7-5dcea933e371 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:40.304193 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a52a5f-ec9c-4216-aad7-5dcea933e371 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:40.325954 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-55a52a5f-ec9c-4216-aad7-5dcea933e371 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:40.327269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-55a52a5f-ec9c-4216-aad7-5dcea933e371 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:40.335581 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1136/2252] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:40.340293 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70be0280-3301-47fb-8af3-b2fb29ac4971 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:40.350912 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70be0280-3301-47fb-8af3-b2fb29ac4971 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:40.356267 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86421953-82bc-4080-bd2f-9eb55071e2ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:40.356843 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86421953-82bc-4080-bd2f-9eb55071e2ec admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:40.357215 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86421953-82bc-4080-bd2f-9eb55071e2ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:40.357432 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86421953-82bc-4080-bd2f-9eb55071e2ec admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:40.358186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86421953-82bc-4080-bd2f-9eb55071e2ec admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:27:40.358390 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86421953-82bc-4080-bd2f-9eb55071e2ec admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:40.360884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86421953-82bc-4080-bd2f-9eb55071e2ec admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:40.379545 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1117/2253] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:27:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:27:40.380780 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a52a5f-ec9c-4216-aad7-5dcea933e371 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:40.394252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a52a5f-ec9c-4216-aad7-5dcea933e371 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:40.413218 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285dd66b-469f-4250-87d3-6996a060bc9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:40.413218 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285dd66b-469f-4250-87d3-6996a060bc9e admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:40.413218 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285dd66b-469f-4250-87d3-6996a060bc9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:40.413218 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285dd66b-469f-4250-87d3-6996a060bc9e admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:40.413527 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285dd66b-469f-4250-87d3-6996a060bc9e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:40.413527 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285dd66b-469f-4250-87d3-6996a060bc9e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:40.444550 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1137/2254] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:40 2022] POST /identity/v3/projects => generated 365 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:40.448931 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86421953-82bc-4080-bd2f-9eb55071e2ec admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:40.463177 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86421953-82bc-4080-bd2f-9eb55071e2ec admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:40.514448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:40.554127 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d0aea8-3ff0-4b12-8e5c-75e4441c8f63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:40.554744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d0aea8-3ff0-4b12-8e5c-75e4441c8f63 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:40.555024 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d0aea8-3ff0-4b12-8e5c-75e4441c8f63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:40.555295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d0aea8-3ff0-4b12-8e5c-75e4441c8f63 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:40.555926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d0aea8-3ff0-4b12-8e5c-75e4441c8f63 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:40.557870 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d0aea8-3ff0-4b12-8e5c-75e4441c8f63 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:40.580971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1118/2255] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:40 2022] POST /identity/v3/users => generated 283 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:40.585529 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285dd66b-469f-4250-87d3-6996a060bc9e admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:40.604228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285dd66b-469f-4250-87d3-6996a060bc9e admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:40.609349 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52c4591-d0c0-49bc-9fe5-4f37de6c9ca9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:40.610840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52c4591-d0c0-49bc-9fe5-4f37de6c9ca9 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:40.611272 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52c4591-d0c0-49bc-9fe5-4f37de6c9ca9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:40.611666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52c4591-d0c0-49bc-9fe5-4f37de6c9ca9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:40.612567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52c4591-d0c0-49bc-9fe5-4f37de6c9ca9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:40.614634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52c4591-d0c0-49bc-9fe5-4f37de6c9ca9 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:40.626951 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1138/2256] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:40 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:40.631129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d0aea8-3ff0-4b12-8e5c-75e4441c8f63 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:40.649246 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d0aea8-3ff0-4b12-8e5c-75e4441c8f63 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:40.649246 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5437026d-b4ee-4ba1-9e3c-e4378bf27f37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:40.649246 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5437026d-b4ee-4ba1-9e3c-e4378bf27f37 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:40.649246 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5437026d-b4ee-4ba1-9e3c-e4378bf27f37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:40.649610 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5437026d-b4ee-4ba1-9e3c-e4378bf27f37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:40.649610 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5437026d-b4ee-4ba1-9e3c-e4378bf27f37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:40.650654 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5437026d-b4ee-4ba1-9e3c-e4378bf27f37 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:40.664149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1119/2257] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:40 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:40.668443 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52c4591-d0c0-49bc-9fe5-4f37de6c9ca9 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:40.683905 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52c4591-d0c0-49bc-9fe5-4f37de6c9ca9 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:40.692586 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53caa825-9168-4a80-bef3-7ceafaa82fd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:40.693511 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53caa825-9168-4a80-bef3-7ceafaa82fd7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:40.693816 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53caa825-9168-4a80-bef3-7ceafaa82fd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:40.694090 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53caa825-9168-4a80-bef3-7ceafaa82fd7 admin admin] PATH_INFO: `/v3/projects/84572fe9e8d44d5a88bcb33ba41cc719/users/f002814967874323b9ab23de40e8b333/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:40.709851 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53caa825-9168-4a80-bef3-7ceafaa82fd7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=84572fe9e8d44d5a88bcb33ba41cc719, user_id=f002814967874323b9ab23de40e8b333, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:40.713667 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53caa825-9168-4a80-bef3-7ceafaa82fd7 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:40.729898 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1139/2258] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:40 2022] PUT /identity/v3/projects/84572fe9e8d44d5a88bcb33ba41cc719/users/f002814967874323b9ab23de40e8b333/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:40.741625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83af6e7c-b55a-4f4e-97fb-431be97afea3 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:40.741909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83af6e7c-b55a-4f4e-97fb-431be97afea3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:40.742179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83af6e7c-b55a-4f4e-97fb-431be97afea3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:40.773345 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-83af6e7c-b55a-4f4e-97fb-431be97afea3 None None] MFA Rules not processed for user `f002814967874323b9ab23de40e8b333`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:40.774850 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-83af6e7c-b55a-4f4e-97fb-431be97afea3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:40.854337 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1120/2259] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:40 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:40.858473 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53caa825-9168-4a80-bef3-7ceafaa82fd7 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:40.924853 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53caa825-9168-4a80-bef3-7ceafaa82fd7 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:40.929597 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccd3d36-fe81-4335-9004-dc6365003362 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:40.930224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccd3d36-fe81-4335-9004-dc6365003362 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:40.930501 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccd3d36-fe81-4335-9004-dc6365003362 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:40.930771 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccd3d36-fe81-4335-9004-dc6365003362 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:40.931414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccd3d36-fe81-4335-9004-dc6365003362 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:40.933357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccd3d36-fe81-4335-9004-dc6365003362 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:40.962024 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1140/2260] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:40 2022] POST /identity/v3/projects => generated 367 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:40.966205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83af6e7c-b55a-4f4e-97fb-431be97afea3 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:40.976655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83af6e7c-b55a-4f4e-97fb-431be97afea3 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:41.033487 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90bbbb30-ff5b-4009-8731-b75493c60892 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:41.034088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bbbb30-ff5b-4009-8731-b75493c60892 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:41.034366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bbbb30-ff5b-4009-8731-b75493c60892 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:41.034636 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bbbb30-ff5b-4009-8731-b75493c60892 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:41.035269 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90bbbb30-ff5b-4009-8731-b75493c60892 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:41.037221 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90bbbb30-ff5b-4009-8731-b75493c60892 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:41.058794 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1121/2261] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:40 2022] POST /identity/v3/users => generated 290 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:41.062750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccd3d36-fe81-4335-9004-dc6365003362 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:41.073959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccd3d36-fe81-4335-9004-dc6365003362 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:41.079478 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d128f3fb-6029-4002-9908-3d2b01124b29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:41.080085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d128f3fb-6029-4002-9908-3d2b01124b29 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:41.080623 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d128f3fb-6029-4002-9908-3d2b01124b29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:41.080908 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d128f3fb-6029-4002-9908-3d2b01124b29 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:41.081582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d128f3fb-6029-4002-9908-3d2b01124b29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:41.083329 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d128f3fb-6029-4002-9908-3d2b01124b29 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:41.091576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:41.103047 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1141/2262] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:41 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:41.103801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90bbbb30-ff5b-4009-8731-b75493c60892 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:41.116704 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90bbbb30-ff5b-4009-8731-b75493c60892 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:41.122468 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a44345c-ec62-407c-9e96-3c41ae0025d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:41.123148 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a44345c-ec62-407c-9e96-3c41ae0025d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:41.123425 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a44345c-ec62-407c-9e96-3c41ae0025d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:41.123696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a44345c-ec62-407c-9e96-3c41ae0025d5 admin admin] PATH_INFO: `/v3/projects/385a6028d5fd42bc8e747b2cb66e5645/users/73c86fcdcd6540338460ffa6471aceb8/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:41.140550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a44345c-ec62-407c-9e96-3c41ae0025d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=385a6028d5fd42bc8e747b2cb66e5645, user_id=73c86fcdcd6540338460ffa6471aceb8, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:41.143243 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a44345c-ec62-407c-9e96-3c41ae0025d5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:41.158909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1122/2263] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:41 2022] PUT /identity/v3/projects/385a6028d5fd42bc8e747b2cb66e5645/users/73c86fcdcd6540338460ffa6471aceb8/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:41.162983 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d128f3fb-6029-4002-9908-3d2b01124b29 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:41.174508 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d128f3fb-6029-4002-9908-3d2b01124b29 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:41.255018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8b1ae6-c2b7-440f-a077-8ff0da37410e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:41.255590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8b1ae6-c2b7-440f-a077-8ff0da37410e admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:41.255911 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8b1ae6-c2b7-440f-a077-8ff0da37410e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:41.256251 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8b1ae6-c2b7-440f-a077-8ff0da37410e admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:41.257074 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8b1ae6-c2b7-440f-a077-8ff0da37410e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:41.258724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8b1ae6-c2b7-440f-a077-8ff0da37410e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:41.286334 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1142/2264] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:41 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:41.286448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a44345c-ec62-407c-9e96-3c41ae0025d5 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:41.293940 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a44345c-ec62-407c-9e96-3c41ae0025d5 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:41.299492 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d733fe8-2bb7-4b91-a6f9-29821a87de8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:41.300153 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d733fe8-2bb7-4b91-a6f9-29821a87de8d admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:41.300439 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d733fe8-2bb7-4b91-a6f9-29821a87de8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:41.300716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d733fe8-2bb7-4b91-a6f9-29821a87de8d admin admin] PATH_INFO: `/v3/projects/385a6028d5fd42bc8e747b2cb66e5645/users/73c86fcdcd6540338460ffa6471aceb8/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:41.302920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d733fe8-2bb7-4b91-a6f9-29821a87de8d admin admin] RBAC: Authorizing `identity:create_grant(project_id=385a6028d5fd42bc8e747b2cb66e5645, user_id=73c86fcdcd6540338460ffa6471aceb8, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:41.305853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d733fe8-2bb7-4b91-a6f9-29821a87de8d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:41.309167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:41.319231 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1123/2265] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:41 2022] PUT /identity/v3/projects/385a6028d5fd42bc8e747b2cb66e5645/users/73c86fcdcd6540338460ffa6471aceb8/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:41.330589 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861eeaeb-bbc3-445e-b826-3e018a547f2f None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:41.330886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861eeaeb-bbc3-445e-b826-3e018a547f2f None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:41.331161 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861eeaeb-bbc3-445e-b826-3e018a547f2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:41.361821 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-861eeaeb-bbc3-445e-b826-3e018a547f2f None None] MFA Rules not processed for user `73c86fcdcd6540338460ffa6471aceb8`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:41.363026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-861eeaeb-bbc3-445e-b826-3e018a547f2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:41.439858 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1143/2266] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:41 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:27:41.462887 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d733fe8-2bb7-4b91-a6f9-29821a87de8d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:41.465100 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3d733fe8-2bb7-4b91-a6f9-29821a87de8d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:41.525085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d733fe8-2bb7-4b91-a6f9-29821a87de8d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:41.532855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c99b04-5249-4123-8d43-6a16b502b277 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:41.533512 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c99b04-5249-4123-8d43-6a16b502b277 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:41.533790 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c99b04-5249-4123-8d43-6a16b502b277 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:41.534060 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c99b04-5249-4123-8d43-6a16b502b277 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:41.544019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c99b04-5249-4123-8d43-6a16b502b277 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:41.546487 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c99b04-5249-4123-8d43-6a16b502b277 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:41.565799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1124/2267] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:41 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:41.736517 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861eeaeb-bbc3-445e-b826-3e018a547f2f None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:41.739332 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-861eeaeb-bbc3-445e-b826-3e018a547f2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:41.751972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861eeaeb-bbc3-445e-b826-3e018a547f2f None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:41.755944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8033dc71-97e2-4dd0-8fc6-93512b88d144 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:41.756630 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8033dc71-97e2-4dd0-8fc6-93512b88d144 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:41.756977 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8033dc71-97e2-4dd0-8fc6-93512b88d144 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:41.757355 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8033dc71-97e2-4dd0-8fc6-93512b88d144 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:41.853457 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8033dc71-97e2-4dd0-8fc6-93512b88d144 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:41.861129 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8033dc71-97e2-4dd0-8fc6-93512b88d144 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:41.881602 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1144/2268] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:41 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:45.244591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c99b04-5249-4123-8d43-6a16b502b277 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:45.246200 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f7c99b04-5249-4123-8d43-6a16b502b277 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:45.341973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c99b04-5249-4123-8d43-6a16b502b277 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:45.346696 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eeff9cf-050e-4441-a6c1-15a8f9e34bf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:45.347323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eeff9cf-050e-4441-a6c1-15a8f9e34bf7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:45.347603 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eeff9cf-050e-4441-a6c1-15a8f9e34bf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:45.347873 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eeff9cf-050e-4441-a6c1-15a8f9e34bf7 admin admin] PATH_INFO: `/v3/users/6130659864db48cf925a1745ef946098` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:45.348991 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eeff9cf-050e-4441-a6c1-15a8f9e34bf7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6130659864db48cf925a1745ef946098)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:45.351370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eeff9cf-050e-4441-a6c1-15a8f9e34bf7 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:45.406516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8eeff9cf-050e-4441-a6c1-15a8f9e34bf7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6130659864db48cf925a1745ef946098'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:45.420257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8eeff9cf-050e-4441-a6c1-15a8f9e34bf7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6130659864db48cf925a1745ef946098'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:45.427292 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8eeff9cf-050e-4441-a6c1-15a8f9e34bf7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6130659864db48cf925a1745ef946098'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:45.435325 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8eeff9cf-050e-4441-a6c1-15a8f9e34bf7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6130659864db48cf925a1745ef946098'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:45.436051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8eeff9cf-050e-4441-a6c1-15a8f9e34bf7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6130659864db48cf925a1745ef946098'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:45.443913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1125/2269] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:45 2022] DELETE /identity/v3/users/6130659864db48cf925a1745ef946098 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:45.449022 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8651f41a-7227-4565-ba7d-e10865749b13 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:45.449570 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8651f41a-7227-4565-ba7d-e10865749b13 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:45.449668 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8651f41a-7227-4565-ba7d-e10865749b13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:45.470027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-8651f41a-7227-4565-ba7d-e10865749b13 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:45.471389 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8651f41a-7227-4565-ba7d-e10865749b13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:45.493724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:45.542941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1145/2270] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:27:45.553509 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eeff9cf-050e-4441-a6c1-15a8f9e34bf7 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:45.555291 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8eeff9cf-050e-4441-a6c1-15a8f9e34bf7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:45.583139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:45.630287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eeff9cf-050e-4441-a6c1-15a8f9e34bf7 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:45.637131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02db33a6-4dcd-4faa-8319-d5eec06cfa45 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:45.637761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02db33a6-4dcd-4faa-8319-d5eec06cfa45 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:45.638045 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02db33a6-4dcd-4faa-8319-d5eec06cfa45 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:45.638319 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02db33a6-4dcd-4faa-8319-d5eec06cfa45 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:45.651594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02db33a6-4dcd-4faa-8319-d5eec06cfa45 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:45.653203 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02db33a6-4dcd-4faa-8319-d5eec06cfa45 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:45.666546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1126/2271] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:27:46.459813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8651f41a-7227-4565-ba7d-e10865749b13 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:46.461984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8651f41a-7227-4565-ba7d-e10865749b13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:46.479860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8651f41a-7227-4565-ba7d-e10865749b13 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:46.497255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3483d2bc-7bc6-4ba8-84a7-eea1a1a2b90c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:46.507756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3483d2bc-7bc6-4ba8-84a7-eea1a1a2b90c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:46.512628 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3483d2bc-7bc6-4ba8-84a7-eea1a1a2b90c admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:46.513512 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3483d2bc-7bc6-4ba8-84a7-eea1a1a2b90c admin admin] PATH_INFO: `/v3/projects/8701651cc71743fbbc9ac74a7aa3845c` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:46.517455 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3483d2bc-7bc6-4ba8-84a7-eea1a1a2b90c admin admin] RBAC: Authorizing `identity:delete_project(project_id=8701651cc71743fbbc9ac74a7aa3845c)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:46.520044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3483d2bc-7bc6-4ba8-84a7-eea1a1a2b90c admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:46.593497 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-3483d2bc-7bc6-4ba8-84a7-eea1a1a2b90c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8701651cc71743fbbc9ac74a7aa3845c', 'request_id': 'req-3483d2bc-7bc6-4ba8-84a7-eea1a1a2b90c'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:46.601681 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-3483d2bc-7bc6-4ba8-84a7-eea1a1a2b90c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8701651cc71743fbbc9ac74a7aa3845c', 'request_id': 'req-3483d2bc-7bc6-4ba8-84a7-eea1a1a2b90c'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:46.611509 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-3483d2bc-7bc6-4ba8-84a7-eea1a1a2b90c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8701651cc71743fbbc9ac74a7aa3845c', 'request_id': 'req-3483d2bc-7bc6-4ba8-84a7-eea1a1a2b90c'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:46.623833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-3483d2bc-7bc6-4ba8-84a7-eea1a1a2b90c admin admin] The token cache is being invalidate because project 8701651cc71743fbbc9ac74a7aa3845c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:46.624272 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-3483d2bc-7bc6-4ba8-84a7-eea1a1a2b90c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:46.625061 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-3483d2bc-7bc6-4ba8-84a7-eea1a1a2b90c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:46.627198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1146/2272] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:46 2022] DELETE /identity/v3/projects/8701651cc71743fbbc9ac74a7aa3845c => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:46.632252 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb856676-b274-44c4-be3a-4f0d2dbeb6c8 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:46.633139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb856676-b274-44c4-be3a-4f0d2dbeb6c8 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:46.633601 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb856676-b274-44c4-be3a-4f0d2dbeb6c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:46.654805 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-cb856676-b274-44c4-be3a-4f0d2dbeb6c8 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:46.656053 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cb856676-b274-44c4-be3a-4f0d2dbeb6c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:46.753588 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1127/2273] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:46.764201 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4caf0d4-9c71-45d0-931c-0a891ca4217f None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:46.764722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4caf0d4-9c71-45d0-931c-0a891ca4217f None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:46.765313 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4caf0d4-9c71-45d0-931c-0a891ca4217f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:46.789442 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-a4caf0d4-9c71-45d0-931c-0a891ca4217f None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:46.791333 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-a4caf0d4-9c71-45d0-931c-0a891ca4217f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:46.807209 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1147/2274] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:46.809627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb856676-b274-44c4-be3a-4f0d2dbeb6c8 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:46.824627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb856676-b274-44c4-be3a-4f0d2dbeb6c8 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:46.831383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3798b11-bd1a-4480-94f8-e8baf48205af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:46.832206 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3798b11-bd1a-4480-94f8-e8baf48205af admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:46.832620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3798b11-bd1a-4480-94f8-e8baf48205af admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:46.832895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3798b11-bd1a-4480-94f8-e8baf48205af admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:46.833630 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3798b11-bd1a-4480-94f8-e8baf48205af admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:27:46.833902 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3798b11-bd1a-4480-94f8-e8baf48205af admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:46.835677 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3798b11-bd1a-4480-94f8-e8baf48205af admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:46.850712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1128/2275] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:27:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:27:46.855614 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4caf0d4-9c71-45d0-931c-0a891ca4217f None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:46.869433 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4caf0d4-9c71-45d0-931c-0a891ca4217f None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:46.878431 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfab86b9-ce9f-4ee9-9bad-2f19e6e0017d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:46.879083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfab86b9-ce9f-4ee9-9bad-2f19e6e0017d admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:46.879363 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfab86b9-ce9f-4ee9-9bad-2f19e6e0017d admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:46.879795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfab86b9-ce9f-4ee9-9bad-2f19e6e0017d admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:46.880677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfab86b9-ce9f-4ee9-9bad-2f19e6e0017d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:46.883471 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfab86b9-ce9f-4ee9-9bad-2f19e6e0017d admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:46.921203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1148/2276] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:46 2022] POST /identity/v3/projects => generated 385 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:46.926106 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3798b11-bd1a-4480-94f8-e8baf48205af admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:46.936947 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3798b11-bd1a-4480-94f8-e8baf48205af admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:46.950302 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfab86b9-ce9f-4ee9-9bad-2f19e6e0017d admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:46.952085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-cfab86b9-ce9f-4ee9-9bad-2f19e6e0017d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:47.006553 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b24a17-159d-4d5b-9e6f-4c6cb17a2975 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:47.007169 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b24a17-159d-4d5b-9e6f-4c6cb17a2975 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:47.007528 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b24a17-159d-4d5b-9e6f-4c6cb17a2975 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:47.007875 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b24a17-159d-4d5b-9e6f-4c6cb17a2975 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:47.008584 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b24a17-159d-4d5b-9e6f-4c6cb17a2975 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:47.010685 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b24a17-159d-4d5b-9e6f-4c6cb17a2975 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:47.028474 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfab86b9-ce9f-4ee9-9bad-2f19e6e0017d admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:47.034872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0afda18-8dd2-4fb6-a0ee-a24d11249878 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:47.035695 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0afda18-8dd2-4fb6-a0ee-a24d11249878 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:47.036139 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0afda18-8dd2-4fb6-a0ee-a24d11249878 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:47.036537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0afda18-8dd2-4fb6-a0ee-a24d11249878 admin admin] PATH_INFO: `/v3/users/f002814967874323b9ab23de40e8b333` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:47.039837 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1129/2277] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:46 2022] POST /identity/v3/users => generated 293 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:47.039964 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0afda18-8dd2-4fb6-a0ee-a24d11249878 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f002814967874323b9ab23de40e8b333)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:47.040660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0afda18-8dd2-4fb6-a0ee-a24d11249878 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:47.040858 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b24a17-159d-4d5b-9e6f-4c6cb17a2975 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:47.053017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b24a17-159d-4d5b-9e6f-4c6cb17a2975 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:47.058308 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453109d5-1a98-4093-beeb-8c445d9bfff4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:47.058954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453109d5-1a98-4093-beeb-8c445d9bfff4 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:47.059287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453109d5-1a98-4093-beeb-8c445d9bfff4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:47.059584 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453109d5-1a98-4093-beeb-8c445d9bfff4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:47.060270 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453109d5-1a98-4093-beeb-8c445d9bfff4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:47.062026 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453109d5-1a98-4093-beeb-8c445d9bfff4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:47.065107 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=90,keystone:DELETE=8,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:47.077454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1130/2278] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:47 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:27:47.081694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453109d5-1a98-4093-beeb-8c445d9bfff4 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:47.093915 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453109d5-1a98-4093-beeb-8c445d9bfff4 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:47.099734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c173657f-02be-4c37-8509-5fdc49aaa778 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:47.100335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c173657f-02be-4c37-8509-5fdc49aaa778 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:47.100609 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c173657f-02be-4c37-8509-5fdc49aaa778 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:47.100875 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c173657f-02be-4c37-8509-5fdc49aaa778 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:47.101550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c173657f-02be-4c37-8509-5fdc49aaa778 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:47.104219 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a0afda18-8dd2-4fb6-a0ee-a24d11249878 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f002814967874323b9ab23de40e8b333'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:47.109352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c173657f-02be-4c37-8509-5fdc49aaa778 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:47.121447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a0afda18-8dd2-4fb6-a0ee-a24d11249878 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f002814967874323b9ab23de40e8b333'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:47.122335 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a0afda18-8dd2-4fb6-a0ee-a24d11249878 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f002814967874323b9ab23de40e8b333'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:47.132071 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1131/2279] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:47 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:27:47.135759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c173657f-02be-4c37-8509-5fdc49aaa778 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:47.140441 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a0afda18-8dd2-4fb6-a0ee-a24d11249878 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f002814967874323b9ab23de40e8b333'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:47.142511 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a0afda18-8dd2-4fb6-a0ee-a24d11249878 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f002814967874323b9ab23de40e8b333'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:47.149463 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c173657f-02be-4c37-8509-5fdc49aaa778 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:47.154987 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475a9684-b7b5-482d-b75d-13c74c8c5340 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:47.155662 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475a9684-b7b5-482d-b75d-13c74c8c5340 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:47.155945 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475a9684-b7b5-482d-b75d-13c74c8c5340 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:47.156259 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475a9684-b7b5-482d-b75d-13c74c8c5340 admin admin] PATH_INFO: `/v3/projects/3a706ffd11144f46ba54becd86464f02/users/5d732bd750a54d8cb516fd5756d64936/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:47.160126 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1149/2280] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:46 2022] DELETE /identity/v3/users/f002814967874323b9ab23de40e8b333 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:47.165305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262ecb38-a3d4-4119-865c-13e222e67125 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:47.165743 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262ecb38-a3d4-4119-865c-13e222e67125 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:47.166180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262ecb38-a3d4-4119-865c-13e222e67125 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:47.182861 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:47.182861 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475a9684-b7b5-482d-b75d-13c74c8c5340 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a706ffd11144f46ba54becd86464f02, user_id=5d732bd750a54d8cb516fd5756d64936, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:47.185927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475a9684-b7b5-482d-b75d-13c74c8c5340 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:47.197844 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-262ecb38-a3d4-4119-865c-13e222e67125 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:47.198721 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-262ecb38-a3d4-4119-865c-13e222e67125 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:47.205147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1132/2281] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:47 2022] PUT /identity/v3/projects/3a706ffd11144f46ba54becd86464f02/users/5d732bd750a54d8cb516fd5756d64936/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:47.207950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec320b6-7c89-4cb2-afed-63e0cb57cb01 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:47.208240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec320b6-7c89-4cb2-afed-63e0cb57cb01 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:47.208512 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec320b6-7c89-4cb2-afed-63e0cb57cb01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:47.241700 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-dec320b6-7c89-4cb2-afed-63e0cb57cb01 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:47.243013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-dec320b6-7c89-4cb2-afed-63e0cb57cb01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:47.283121 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1150/2282] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:27:47.294515 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262ecb38-a3d4-4119-865c-13e222e67125 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:47.296062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-262ecb38-a3d4-4119-865c-13e222e67125 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:47.338570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1133/2283] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:27:47.350162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec320b6-7c89-4cb2-afed-63e0cb57cb01 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:47.351650 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-dec320b6-7c89-4cb2-afed-63e0cb57cb01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:47.441729 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262ecb38-a3d4-4119-865c-13e222e67125 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:47.456555 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4e891a-c7a3-4f1f-9e02-24799d41763f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:47.457414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4e891a-c7a3-4f1f-9e02-24799d41763f service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:47.457704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4e891a-c7a3-4f1f-9e02-24799d41763f service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:47.457985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4e891a-c7a3-4f1f-9e02-24799d41763f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:47.464612 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec320b6-7c89-4cb2-afed-63e0cb57cb01 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:47.473365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde9194a-d29f-41a8-b789-71c43fff25e1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:47.474485 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde9194a-d29f-41a8-b789-71c43fff25e1 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:47.474791 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde9194a-d29f-41a8-b789-71c43fff25e1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:47.475069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde9194a-d29f-41a8-b789-71c43fff25e1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:47.482390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4e891a-c7a3-4f1f-9e02-24799d41763f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:47.493976 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4e891a-c7a3-4f1f-9e02-24799d41763f service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:47.501190 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde9194a-d29f-41a8-b789-71c43fff25e1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:47.502669 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde9194a-d29f-41a8-b789-71c43fff25e1 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:47.505020 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1151/2284] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:47.522906 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1134/2285] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:48.176981 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4e891a-c7a3-4f1f-9e02-24799d41763f service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:48.180583 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-2f4e891a-c7a3-4f1f-9e02-24799d41763f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:48.204549 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4e891a-c7a3-4f1f-9e02-24799d41763f service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:48.209577 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d90ef8f-a2ef-4ecd-9219-42601f4f1036 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:48.210380 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d90ef8f-a2ef-4ecd-9219-42601f4f1036 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:48.210663 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d90ef8f-a2ef-4ecd-9219-42601f4f1036 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:48.211068 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d90ef8f-a2ef-4ecd-9219-42601f4f1036 admin admin] PATH_INFO: `/v3/projects/84572fe9e8d44d5a88bcb33ba41cc719` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:48.212213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d90ef8f-a2ef-4ecd-9219-42601f4f1036 admin admin] RBAC: Authorizing `identity:delete_project(project_id=84572fe9e8d44d5a88bcb33ba41cc719)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:48.214245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d90ef8f-a2ef-4ecd-9219-42601f4f1036 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:48.260881 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:48.280580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-8d90ef8f-a2ef-4ecd-9219-42601f4f1036 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '84572fe9e8d44d5a88bcb33ba41cc719', 'request_id': 'req-8d90ef8f-a2ef-4ecd-9219-42601f4f1036'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:48.286504 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-8d90ef8f-a2ef-4ecd-9219-42601f4f1036 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '84572fe9e8d44d5a88bcb33ba41cc719', 'request_id': 'req-8d90ef8f-a2ef-4ecd-9219-42601f4f1036'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:48.297858 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-8d90ef8f-a2ef-4ecd-9219-42601f4f1036 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '84572fe9e8d44d5a88bcb33ba41cc719', 'request_id': 'req-8d90ef8f-a2ef-4ecd-9219-42601f4f1036'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:48.317491 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-8d90ef8f-a2ef-4ecd-9219-42601f4f1036 admin admin] The token cache is being invalidate because project 84572fe9e8d44d5a88bcb33ba41cc719 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:48.317806 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-8d90ef8f-a2ef-4ecd-9219-42601f4f1036 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:48.318437 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-8d90ef8f-a2ef-4ecd-9219-42601f4f1036 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:48.323793 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1152/2286] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:48 2022] DELETE /identity/v3/projects/84572fe9e8d44d5a88bcb33ba41cc719 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:48.324622 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde9194a-d29f-41a8-b789-71c43fff25e1 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:48.338047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde9194a-d29f-41a8-b789-71c43fff25e1 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:48.406581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48265bad-055b-4ad7-be22-d813e00d5a50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:48.407249 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48265bad-055b-4ad7-be22-d813e00d5a50 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:48.407522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48265bad-055b-4ad7-be22-d813e00d5a50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:48.407809 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48265bad-055b-4ad7-be22-d813e00d5a50 admin admin] PATH_INFO: `/v3/users/73c86fcdcd6540338460ffa6471aceb8` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:48.408900 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48265bad-055b-4ad7-be22-d813e00d5a50 admin admin] RBAC: Authorizing `identity:delete_user(user_id=73c86fcdcd6540338460ffa6471aceb8)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:48.410853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48265bad-055b-4ad7-be22-d813e00d5a50 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:48.459831 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=94,keystone:INSERT=1,keystone:DELETE=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:48.472735 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-48265bad-055b-4ad7-be22-d813e00d5a50 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '73c86fcdcd6540338460ffa6471aceb8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:48.487740 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-48265bad-055b-4ad7-be22-d813e00d5a50 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '73c86fcdcd6540338460ffa6471aceb8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:48.496933 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-48265bad-055b-4ad7-be22-d813e00d5a50 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '73c86fcdcd6540338460ffa6471aceb8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:48.507214 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-48265bad-055b-4ad7-be22-d813e00d5a50 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '73c86fcdcd6540338460ffa6471aceb8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:48.508088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-48265bad-055b-4ad7-be22-d813e00d5a50 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '73c86fcdcd6540338460ffa6471aceb8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:48.516763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1135/2287] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:48 2022] DELETE /identity/v3/users/73c86fcdcd6540338460ffa6471aceb8 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:49.248612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d90ef8f-a2ef-4ecd-9219-42601f4f1036 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:49.249813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8d90ef8f-a2ef-4ecd-9219-42601f4f1036 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:49.262804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d90ef8f-a2ef-4ecd-9219-42601f4f1036 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:49.374761 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce407d9d-414a-4e4e-b8a0-5f8cee95750f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:49.374761 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce407d9d-414a-4e4e-b8a0-5f8cee95750f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:49.374761 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce407d9d-414a-4e4e-b8a0-5f8cee95750f admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:49.375063 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce407d9d-414a-4e4e-b8a0-5f8cee95750f admin admin] PATH_INFO: `/v3/projects/385a6028d5fd42bc8e747b2cb66e5645` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:49.376403 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce407d9d-414a-4e4e-b8a0-5f8cee95750f admin admin] RBAC: Authorizing `identity:delete_project(project_id=385a6028d5fd42bc8e747b2cb66e5645)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:49.381484 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce407d9d-414a-4e4e-b8a0-5f8cee95750f admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:49.457209 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ce407d9d-414a-4e4e-b8a0-5f8cee95750f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '385a6028d5fd42bc8e747b2cb66e5645', 'request_id': 'req-ce407d9d-414a-4e4e-b8a0-5f8cee95750f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:49.467516 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ce407d9d-414a-4e4e-b8a0-5f8cee95750f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '385a6028d5fd42bc8e747b2cb66e5645', 'request_id': 'req-ce407d9d-414a-4e4e-b8a0-5f8cee95750f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:49.479090 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ce407d9d-414a-4e4e-b8a0-5f8cee95750f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '385a6028d5fd42bc8e747b2cb66e5645', 'request_id': 'req-ce407d9d-414a-4e4e-b8a0-5f8cee95750f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:49.512136 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ce407d9d-414a-4e4e-b8a0-5f8cee95750f admin admin] The token cache is being invalidate because project 385a6028d5fd42bc8e747b2cb66e5645 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:49.512487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ce407d9d-414a-4e4e-b8a0-5f8cee95750f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:49.514038 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ce407d9d-414a-4e4e-b8a0-5f8cee95750f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:49.515827 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1153/2288] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:49 2022] DELETE /identity/v3/projects/385a6028d5fd42bc8e747b2cb66e5645 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:49.521565 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba671aa-6141-4df5-a0c4-9ba9f292df58 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:49.522067 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba671aa-6141-4df5-a0c4-9ba9f292df58 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:49.522163 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba671aa-6141-4df5-a0c4-9ba9f292df58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:49.551508 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-6ba671aa-6141-4df5-a0c4-9ba9f292df58 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:49.552898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6ba671aa-6141-4df5-a0c4-9ba9f292df58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:49.666425 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1136/2289] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:27:49.673902 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae53b00c-78a6-4743-95ec-2c47b5ffa4c3 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:49.674316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae53b00c-78a6-4743-95ec-2c47b5ffa4c3 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:49.674696 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae53b00c-78a6-4743-95ec-2c47b5ffa4c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:49.699050 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-ae53b00c-78a6-4743-95ec-2c47b5ffa4c3 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:49.700612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ae53b00c-78a6-4743-95ec-2c47b5ffa4c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:49.711371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1154/2290] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:49.717570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba671aa-6141-4df5-a0c4-9ba9f292df58 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:49.730728 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba671aa-6141-4df5-a0c4-9ba9f292df58 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:49.738967 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0c6f91-a1bd-4c73-9206-39f534ebeb47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:49.739701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0c6f91-a1bd-4c73-9206-39f534ebeb47 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:49.740063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0c6f91-a1bd-4c73-9206-39f534ebeb47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:49.740357 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0c6f91-a1bd-4c73-9206-39f534ebeb47 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:49.741313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0c6f91-a1bd-4c73-9206-39f534ebeb47 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:27:49.741677 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0c6f91-a1bd-4c73-9206-39f534ebeb47 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:49.744080 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0c6f91-a1bd-4c73-9206-39f534ebeb47 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:49.768093 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1137/2291] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:27:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:27:49.771325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae53b00c-78a6-4743-95ec-2c47b5ffa4c3 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:49.791643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae53b00c-78a6-4743-95ec-2c47b5ffa4c3 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:49.803365 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923b007c-02dc-40f6-a0e8-69a4afb7b19c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:49.804047 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923b007c-02dc-40f6-a0e8-69a4afb7b19c admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:49.804503 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923b007c-02dc-40f6-a0e8-69a4afb7b19c admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:49.804782 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923b007c-02dc-40f6-a0e8-69a4afb7b19c admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:49.805463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-923b007c-02dc-40f6-a0e8-69a4afb7b19c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:49.808970 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-923b007c-02dc-40f6-a0e8-69a4afb7b19c admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:49.861225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1155/2292] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:49 2022] POST /identity/v3/projects => generated 383 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:49.863820 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0c6f91-a1bd-4c73-9206-39f534ebeb47 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:49.879735 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0c6f91-a1bd-4c73-9206-39f534ebeb47 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:50.023900 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501e771e-bb6f-47a6-8ab7-84cc26c381f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:50.024534 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501e771e-bb6f-47a6-8ab7-84cc26c381f6 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:50.024795 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501e771e-bb6f-47a6-8ab7-84cc26c381f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:50.025117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501e771e-bb6f-47a6-8ab7-84cc26c381f6 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:50.025714 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501e771e-bb6f-47a6-8ab7-84cc26c381f6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:50.027706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501e771e-bb6f-47a6-8ab7-84cc26c381f6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:50.095271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1138/2293] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:49 2022] POST /identity/v3/users => generated 292 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:50.099660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923b007c-02dc-40f6-a0e8-69a4afb7b19c admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:50.119324 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923b007c-02dc-40f6-a0e8-69a4afb7b19c admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:50.130576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ea9a0a-ed75-4b3c-b6b2-5a4a9f19cc9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:50.131212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ea9a0a-ed75-4b3c-b6b2-5a4a9f19cc9c admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:50.131493 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ea9a0a-ed75-4b3c-b6b2-5a4a9f19cc9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:50.131768 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ea9a0a-ed75-4b3c-b6b2-5a4a9f19cc9c admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:50.132701 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ea9a0a-ed75-4b3c-b6b2-5a4a9f19cc9c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:50.139077 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ea9a0a-ed75-4b3c-b6b2-5a4a9f19cc9c admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:50.160828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1156/2294] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:50 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:27:50.169656 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501e771e-bb6f-47a6-8ab7-84cc26c381f6 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:50.185753 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501e771e-bb6f-47a6-8ab7-84cc26c381f6 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:50.198365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23966f1d-d9f8-4eb2-995e-a8b2c9785d45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:50.198989 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23966f1d-d9f8-4eb2-995e-a8b2c9785d45 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:50.199269 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23966f1d-d9f8-4eb2-995e-a8b2c9785d45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:50.199537 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23966f1d-d9f8-4eb2-995e-a8b2c9785d45 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:50.200710 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23966f1d-d9f8-4eb2-995e-a8b2c9785d45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:50.202383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23966f1d-d9f8-4eb2-995e-a8b2c9785d45 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:50.238228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1139/2295] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:50 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:50.239934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ea9a0a-ed75-4b3c-b6b2-5a4a9f19cc9c admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:50.254557 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ea9a0a-ed75-4b3c-b6b2-5a4a9f19cc9c admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:50.262923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3aff44-30c8-4573-90c4-f40c70536bd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:50.263782 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3aff44-30c8-4573-90c4-f40c70536bd4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:50.264207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3aff44-30c8-4573-90c4-f40c70536bd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:50.264864 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3aff44-30c8-4573-90c4-f40c70536bd4 admin admin] PATH_INFO: `/v3/projects/58ddc9e9429740e391a4e5399111514a/users/5a4ddc10cf85459ca0cc876c22b4d2aa/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:50.276712 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:50.303093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3aff44-30c8-4573-90c4-f40c70536bd4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=58ddc9e9429740e391a4e5399111514a, user_id=5a4ddc10cf85459ca0cc876c22b4d2aa, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:50.306589 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3aff44-30c8-4573-90c4-f40c70536bd4 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:50.337095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1157/2296] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:50 2022] PUT /identity/v3/projects/58ddc9e9429740e391a4e5399111514a/users/5a4ddc10cf85459ca0cc876c22b4d2aa/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:50.346896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4f5e79-27be-4d8c-bab8-4afd6eafe314 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:50.347291 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4f5e79-27be-4d8c-bab8-4afd6eafe314 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:50.347665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4f5e79-27be-4d8c-bab8-4afd6eafe314 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:50.416363 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-2d4f5e79-27be-4d8c-bab8-4afd6eafe314 None None] MFA Rules not processed for user `5a4ddc10cf85459ca0cc876c22b4d2aa`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:50.417764 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2d4f5e79-27be-4d8c-bab8-4afd6eafe314 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:50.444529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:50.582852 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3aff44-30c8-4573-90c4-f40c70536bd4 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:50.584176 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1140/2297] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:50 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:50.743140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3aff44-30c8-4573-90c4-f40c70536bd4 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:50.754338 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6779a4-d950-4b39-bf9c-b118a438a100 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:50.755595 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6779a4-d950-4b39-bf9c-b118a438a100 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:50.756024 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6779a4-d950-4b39-bf9c-b118a438a100 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:50.756424 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6779a4-d950-4b39-bf9c-b118a438a100 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:50.757576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6779a4-d950-4b39-bf9c-b118a438a100 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:50.760106 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6779a4-d950-4b39-bf9c-b118a438a100 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:50.809925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1158/2298] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:50 2022] POST /identity/v3/projects => generated 383 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:27:50.815012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4f5e79-27be-4d8c-bab8-4afd6eafe314 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:50.838012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4f5e79-27be-4d8c-bab8-4afd6eafe314 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:50.931315 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ddcf3f-6ed1-4c24-bd3c-3858be4b5eca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:50.932254 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ddcf3f-6ed1-4c24-bd3c-3858be4b5eca admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:50.932670 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ddcf3f-6ed1-4c24-bd3c-3858be4b5eca admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:50.933256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ddcf3f-6ed1-4c24-bd3c-3858be4b5eca admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:50.934113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ddcf3f-6ed1-4c24-bd3c-3858be4b5eca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:50.938733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ddcf3f-6ed1-4c24-bd3c-3858be4b5eca admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:50.965900 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1141/2299] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:50 2022] POST /identity/v3/users => generated 292 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:50.971148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6779a4-d950-4b39-bf9c-b118a438a100 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:50.988383 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6779a4-d950-4b39-bf9c-b118a438a100 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:50.993702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c82072c-0bd2-4ab9-82a7-d8f983d2b12d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:50.994328 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c82072c-0bd2-4ab9-82a7-d8f983d2b12d admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:50.994756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c82072c-0bd2-4ab9-82a7-d8f983d2b12d admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:50.995035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c82072c-0bd2-4ab9-82a7-d8f983d2b12d admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:50.996121 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c82072c-0bd2-4ab9-82a7-d8f983d2b12d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:50.997957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c82072c-0bd2-4ab9-82a7-d8f983d2b12d admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:51.019868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1159/2300] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:50 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:51.021179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ddcf3f-6ed1-4c24-bd3c-3858be4b5eca admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:51.038048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ddcf3f-6ed1-4c24-bd3c-3858be4b5eca admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:51.048910 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33222ef5-bbce-4f73-bbc6-201947f2cc1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:51.049798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33222ef5-bbce-4f73-bbc6-201947f2cc1f admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:51.050167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33222ef5-bbce-4f73-bbc6-201947f2cc1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:51.050539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33222ef5-bbce-4f73-bbc6-201947f2cc1f admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:51.051285 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33222ef5-bbce-4f73-bbc6-201947f2cc1f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:51.053084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33222ef5-bbce-4f73-bbc6-201947f2cc1f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:51.072961 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1142/2301] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:51 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:51.076347 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c82072c-0bd2-4ab9-82a7-d8f983d2b12d admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:51.099951 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c82072c-0bd2-4ab9-82a7-d8f983d2b12d admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:51.112157 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd0101a-9c7d-477a-86dd-d37ac3a90209 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:51.117473 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd0101a-9c7d-477a-86dd-d37ac3a90209 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:51.117775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd0101a-9c7d-477a-86dd-d37ac3a90209 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:51.118299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd0101a-9c7d-477a-86dd-d37ac3a90209 admin admin] PATH_INFO: `/v3/projects/dda000b6a4464d89b8e86f754cac24c2/users/0c7ba20acff948db9ecc494b0ae12a55/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:51.141956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd0101a-9c7d-477a-86dd-d37ac3a90209 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dda000b6a4464d89b8e86f754cac24c2, user_id=0c7ba20acff948db9ecc494b0ae12a55, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:51.144656 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd0101a-9c7d-477a-86dd-d37ac3a90209 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:51.168041 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1160/2302] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:51 2022] PUT /identity/v3/projects/dda000b6a4464d89b8e86f754cac24c2/users/0c7ba20acff948db9ecc494b0ae12a55/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:51.179980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6294d43-cf3a-4615-a06c-b570b5dedb21 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:51.180245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6294d43-cf3a-4615-a06c-b570b5dedb21 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:51.180515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6294d43-cf3a-4615-a06c-b570b5dedb21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:51.233182 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-d6294d43-cf3a-4615-a06c-b570b5dedb21 None None] MFA Rules not processed for user `0c7ba20acff948db9ecc494b0ae12a55`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:51.234675 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d6294d43-cf3a-4615-a06c-b570b5dedb21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:51.302332 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd0101a-9c7d-477a-86dd-d37ac3a90209 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:51.304528 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-bcd0101a-9c7d-477a-86dd-d37ac3a90209 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:51.414430 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd0101a-9c7d-477a-86dd-d37ac3a90209 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:51.417350 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1143/2303] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:51 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:51.421975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6294d43-cf3a-4615-a06c-b570b5dedb21 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:51.432704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c66c1d-89a3-42b4-8ec1-1a38eb6a9a13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:51.433448 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c66c1d-89a3-42b4-8ec1-1a38eb6a9a13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:51.433677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c66c1d-89a3-42b4-8ec1-1a38eb6a9a13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:51.433950 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c66c1d-89a3-42b4-8ec1-1a38eb6a9a13 admin admin] PATH_INFO: `/v3/users/298b05a8580946f4ab2ed66f0b9a3c2b` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:51.435847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c66c1d-89a3-42b4-8ec1-1a38eb6a9a13 admin admin] RBAC: Authorizing `identity:delete_user(user_id=298b05a8580946f4ab2ed66f0b9a3c2b)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:51.437894 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c66c1d-89a3-42b4-8ec1-1a38eb6a9a13 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:51.457329 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6294d43-cf3a-4615-a06c-b570b5dedb21 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:51.463215 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071cbc96-912a-455c-a42d-40fde2691bb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:51.464550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071cbc96-912a-455c-a42d-40fde2691bb6 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:51.465221 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071cbc96-912a-455c-a42d-40fde2691bb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:51.465562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071cbc96-912a-455c-a42d-40fde2691bb6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:51.466404 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071cbc96-912a-455c-a42d-40fde2691bb6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:51.469814 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071cbc96-912a-455c-a42d-40fde2691bb6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:51.502803 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:51.531088 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:51.544859 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-48c66c1d-89a3-42b4-8ec1-1a38eb6a9a13 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '298b05a8580946f4ab2ed66f0b9a3c2b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:51.551799 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1144/2304] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:51 2022] POST /identity/v3/projects => generated 383 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:27:51.552877 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071cbc96-912a-455c-a42d-40fde2691bb6 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:51.556993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-48c66c1d-89a3-42b4-8ec1-1a38eb6a9a13 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '298b05a8580946f4ab2ed66f0b9a3c2b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:51.558224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-48c66c1d-89a3-42b4-8ec1-1a38eb6a9a13 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '298b05a8580946f4ab2ed66f0b9a3c2b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:51.584970 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-48c66c1d-89a3-42b4-8ec1-1a38eb6a9a13 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '298b05a8580946f4ab2ed66f0b9a3c2b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:51.585939 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-48c66c1d-89a3-42b4-8ec1-1a38eb6a9a13 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '298b05a8580946f4ab2ed66f0b9a3c2b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:51.613502 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1161/2305] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:27:51 2022] DELETE /identity/v3/users/298b05a8580946f4ab2ed66f0b9a3c2b => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:51.680327 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071cbc96-912a-455c-a42d-40fde2691bb6 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:51.818347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191acc3f-dac0-4fc8-9de3-798b545e6f9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:51.820191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191acc3f-dac0-4fc8-9de3-798b545e6f9d admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:51.820547 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191acc3f-dac0-4fc8-9de3-798b545e6f9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:51.820898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191acc3f-dac0-4fc8-9de3-798b545e6f9d admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:51.821732 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191acc3f-dac0-4fc8-9de3-798b545e6f9d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:51.826969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191acc3f-dac0-4fc8-9de3-798b545e6f9d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:51.867687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c66c1d-89a3-42b4-8ec1-1a38eb6a9a13 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:51.869458 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-48c66c1d-89a3-42b4-8ec1-1a38eb6a9a13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:51.889110 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1145/2306] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:51 2022] POST /identity/v3/users => generated 298 bytes in 337 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:51.892648 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c66c1d-89a3-42b4-8ec1-1a38eb6a9a13 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:51.894126 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191acc3f-dac0-4fc8-9de3-798b545e6f9d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:51.896199 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-191acc3f-dac0-4fc8-9de3-798b545e6f9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:51.897781 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd593b25-4f1a-4756-ab77-a0974650fdfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:51.898567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd593b25-4f1a-4756-ab77-a0974650fdfb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:51.898798 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd593b25-4f1a-4756-ab77-a0974650fdfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:51.899012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd593b25-4f1a-4756-ab77-a0974650fdfb admin admin] PATH_INFO: `/v3/projects/959853ec613f47819ed2883c8539b24e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:51.900162 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd593b25-4f1a-4756-ab77-a0974650fdfb admin admin] RBAC: Authorizing `identity:delete_project(project_id=959853ec613f47819ed2883c8539b24e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:51.902257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd593b25-4f1a-4756-ab77-a0974650fdfb admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:51.920281 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191acc3f-dac0-4fc8-9de3-798b545e6f9d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:51.934798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8ed4ea-4705-4806-ba00-96d026249bf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:51.935433 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8ed4ea-4705-4806-ba00-96d026249bf4 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:51.935764 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8ed4ea-4705-4806-ba00-96d026249bf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:51.935987 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8ed4ea-4705-4806-ba00-96d026249bf4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:51.936649 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8ed4ea-4705-4806-ba00-96d026249bf4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:51.938808 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8ed4ea-4705-4806-ba00-96d026249bf4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:51.969332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1146/2307] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:51 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:27:51.977455 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8ed4ea-4705-4806-ba00-96d026249bf4 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:51.995791 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8ed4ea-4705-4806-ba00-96d026249bf4 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:52.011138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bd593b25-4f1a-4756-ab77-a0974650fdfb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '959853ec613f47819ed2883c8539b24e', 'request_id': 'req-bd593b25-4f1a-4756-ab77-a0974650fdfb'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:52.019591 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bd593b25-4f1a-4756-ab77-a0974650fdfb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '959853ec613f47819ed2883c8539b24e', 'request_id': 'req-bd593b25-4f1a-4756-ab77-a0974650fdfb'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:52.035303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bd593b25-4f1a-4756-ab77-a0974650fdfb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '959853ec613f47819ed2883c8539b24e', 'request_id': 'req-bd593b25-4f1a-4756-ab77-a0974650fdfb'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:52.057915 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bd593b25-4f1a-4756-ab77-a0974650fdfb admin admin] The token cache is being invalidate because project 959853ec613f47819ed2883c8539b24e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:27:52.058289 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bd593b25-4f1a-4756-ab77-a0974650fdfb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:52.059467 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bd593b25-4f1a-4756-ab77-a0974650fdfb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:27:52.065827 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1162/2308] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:27:51 2022] DELETE /identity/v3/projects/959853ec613f47819ed2883c8539b24e => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:52.068195 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebcf495-3842-4098-abc6-6d8711406e50 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:52.068195 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebcf495-3842-4098-abc6-6d8711406e50 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:52.068195 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebcf495-3842-4098-abc6-6d8711406e50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:52.107248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-8ebcf495-3842-4098-abc6-6d8711406e50 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:52.108508 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8ebcf495-3842-4098-abc6-6d8711406e50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:52.142525 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e257aec-3daf-4db5-9cd9-c54ebaeb0e40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:52.143678 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e257aec-3daf-4db5-9cd9-c54ebaeb0e40 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:52.143975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e257aec-3daf-4db5-9cd9-c54ebaeb0e40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:52.144258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e257aec-3daf-4db5-9cd9-c54ebaeb0e40 admin admin] PATH_INFO: `/v3/projects/9eae429122fa4946b3f56abcc7e7cfd8/users/c373ee5e1669473aa111280fb5b595cb/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:52.189942 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e257aec-3daf-4db5-9cd9-c54ebaeb0e40 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9eae429122fa4946b3f56abcc7e7cfd8, user_id=c373ee5e1669473aa111280fb5b595cb, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:52.193076 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e257aec-3daf-4db5-9cd9-c54ebaeb0e40 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:52.224210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1147/2309] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:51 2022] PUT /identity/v3/projects/9eae429122fa4946b3f56abcc7e7cfd8/users/c373ee5e1669473aa111280fb5b595cb/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:52.228721 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e257aec-3daf-4db5-9cd9-c54ebaeb0e40 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:52.230804 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5e257aec-3daf-4db5-9cd9-c54ebaeb0e40 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:52.238586 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:52.293286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1163/2310] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:27:52.302330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c572afd2-62ab-4208-b9cb-9a02ffe98671 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:52.302765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c572afd2-62ab-4208-b9cb-9a02ffe98671 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:52.303311 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c572afd2-62ab-4208-b9cb-9a02ffe98671 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:52.338155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-c572afd2-62ab-4208-b9cb-9a02ffe98671 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:52.339943 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c572afd2-62ab-4208-b9cb-9a02ffe98671 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:52.354307 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1164/2311] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:27:52.358619 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c572afd2-62ab-4208-b9cb-9a02ffe98671 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:52.365461 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e257aec-3daf-4db5-9cd9-c54ebaeb0e40 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:52.371449 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ac963e-db02-4d25-8f3a-1a7900435f40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:52.372896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ac963e-db02-4d25-8f3a-1a7900435f40 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:52.373352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ac963e-db02-4d25-8f3a-1a7900435f40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:52.373769 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ac963e-db02-4d25-8f3a-1a7900435f40 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:52.374579 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ac963e-db02-4d25-8f3a-1a7900435f40 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:52.374752 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c572afd2-62ab-4208-b9cb-9a02ffe98671 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:52.377640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ac963e-db02-4d25-8f3a-1a7900435f40 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:52.383729 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45794ef-2818-4851-8803-18c0b27855d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:52.387456 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45794ef-2818-4851-8803-18c0b27855d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:52.387456 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45794ef-2818-4851-8803-18c0b27855d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:52.387456 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45794ef-2818-4851-8803-18c0b27855d8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:52.388478 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45794ef-2818-4851-8803-18c0b27855d8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:27:52.389850 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45794ef-2818-4851-8803-18c0b27855d8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:52.393744 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45794ef-2818-4851-8803-18c0b27855d8 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:52.395062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1148/2312] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:52 2022] GET /identity/v3/roles => generated 1420 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:52.398992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ac963e-db02-4d25-8f3a-1a7900435f40 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:52.412143 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1165/2313] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:27:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:27:52.412343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ac963e-db02-4d25-8f3a-1a7900435f40 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:52.414070 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45794ef-2818-4851-8803-18c0b27855d8 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:52.416855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99bb15f-08a4-4bcb-8eb1-96b0260c4c56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:52.417686 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99bb15f-08a4-4bcb-8eb1-96b0260c4c56 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:52.418173 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99bb15f-08a4-4bcb-8eb1-96b0260c4c56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:52.418553 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99bb15f-08a4-4bcb-8eb1-96b0260c4c56 admin admin] PATH_INFO: `/v3/projects/9eae429122fa4946b3f56abcc7e7cfd8/users/c373ee5e1669473aa111280fb5b595cb/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:52.421355 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99bb15f-08a4-4bcb-8eb1-96b0260c4c56 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9eae429122fa4946b3f56abcc7e7cfd8, user_id=c373ee5e1669473aa111280fb5b595cb, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:52.425347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99bb15f-08a4-4bcb-8eb1-96b0260c4c56 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:52.437496 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45794ef-2818-4851-8803-18c0b27855d8 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:52.450526 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1149/2314] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:52 2022] PUT /identity/v3/projects/9eae429122fa4946b3f56abcc7e7cfd8/users/c373ee5e1669473aa111280fb5b595cb/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:52.461811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7002904c-e4d2-4a9d-8675-b6aed0ca7928 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:52.462101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7002904c-e4d2-4a9d-8675-b6aed0ca7928 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:52.462379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7002904c-e4d2-4a9d-8675-b6aed0ca7928 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:52.478659 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:52.514872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-7002904c-e4d2-4a9d-8675-b6aed0ca7928 None None] MFA Rules not processed for user `c373ee5e1669473aa111280fb5b595cb`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:52.516766 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7002904c-e4d2-4a9d-8675-b6aed0ca7928 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:52.540409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d306ea-02a5-4d66-b6a4-7da3f457d97f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:52.541472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d306ea-02a5-4d66-b6a4-7da3f457d97f admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:52.541596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d306ea-02a5-4d66-b6a4-7da3f457d97f admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:52.541962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d306ea-02a5-4d66-b6a4-7da3f457d97f admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:52.542705 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d306ea-02a5-4d66-b6a4-7da3f457d97f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:52.545203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d306ea-02a5-4d66-b6a4-7da3f457d97f admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:52.613414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1166/2315] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:52 2022] POST /identity/v3/projects => generated 387 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:52.617763 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d306ea-02a5-4d66-b6a4-7da3f457d97f admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:52.640068 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1150/2316] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:52 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:27:52.652887 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7002904c-e4d2-4a9d-8675-b6aed0ca7928 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:52.659247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7002904c-e4d2-4a9d-8675-b6aed0ca7928 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:52.733168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d306ea-02a5-4d66-b6a4-7da3f457d97f admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:52.740801 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16bfdbd-2036-4f7a-8946-9d988c577c4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:52.741501 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16bfdbd-2036-4f7a-8946-9d988c577c4c admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:52.741863 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16bfdbd-2036-4f7a-8946-9d988c577c4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:52.742137 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16bfdbd-2036-4f7a-8946-9d988c577c4c admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:52.743925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16bfdbd-2036-4f7a-8946-9d988c577c4c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:52.745778 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16bfdbd-2036-4f7a-8946-9d988c577c4c admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:52.751807 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7002904c-e4d2-4a9d-8675-b6aed0ca7928 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:52.774653 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1167/2317] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:52 2022] POST /identity/v3/users => generated 294 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:52.781280 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16bfdbd-2036-4f7a-8946-9d988c577c4c admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:52.796759 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16bfdbd-2036-4f7a-8946-9d988c577c4c admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:52.802805 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:52.803172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351583ef-cde1-4ead-8e48-d08187352cbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:52.803803 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351583ef-cde1-4ead-8e48-d08187352cbf admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:52.804106 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351583ef-cde1-4ead-8e48-d08187352cbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:52.804356 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351583ef-cde1-4ead-8e48-d08187352cbf admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:52.805060 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351583ef-cde1-4ead-8e48-d08187352cbf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:52.809136 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351583ef-cde1-4ead-8e48-d08187352cbf admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:52.821793 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1168/2318] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:52 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:52.825758 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351583ef-cde1-4ead-8e48-d08187352cbf admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:52.840582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351583ef-cde1-4ead-8e48-d08187352cbf admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:52.846355 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45f97f8-924f-4f53-8890-151e437b3720 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:52.847173 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f59883-e20f-41ba-8e58-1383e808edd2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:52.847310 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45f97f8-924f-4f53-8890-151e437b3720 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:52.847643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45f97f8-924f-4f53-8890-151e437b3720 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:52.847982 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45f97f8-924f-4f53-8890-151e437b3720 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:52.848254 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f59883-e20f-41ba-8e58-1383e808edd2 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:52.848553 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f59883-e20f-41ba-8e58-1383e808edd2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:52.848824 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f59883-e20f-41ba-8e58-1383e808edd2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:52.849054 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45f97f8-924f-4f53-8890-151e437b3720 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:52.850988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45f97f8-924f-4f53-8890-151e437b3720 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:52.851726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-26f59883-e20f-41ba-8e58-1383e808edd2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:52.864164 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1169/2319] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:52 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:27:52.868674 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45f97f8-924f-4f53-8890-151e437b3720 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:52.894283 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45f97f8-924f-4f53-8890-151e437b3720 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:52.901338 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38786606-669b-4a98-98f5-c8b4fee89c10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:52.902321 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38786606-669b-4a98-98f5-c8b4fee89c10 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:52.903714 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38786606-669b-4a98-98f5-c8b4fee89c10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:52.904171 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38786606-669b-4a98-98f5-c8b4fee89c10 admin admin] PATH_INFO: `/v3/projects/8afb4049582c4beaa9defa2fe20c8b52/users/b05dcf62adc949c79b37c92bb3e09e8e/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:52.925459 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38786606-669b-4a98-98f5-c8b4fee89c10 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8afb4049582c4beaa9defa2fe20c8b52, user_id=b05dcf62adc949c79b37c92bb3e09e8e, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:52.928907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38786606-669b-4a98-98f5-c8b4fee89c10 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:52.961363 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1170/2320] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:52 2022] PUT /identity/v3/projects/8afb4049582c4beaa9defa2fe20c8b52/users/b05dcf62adc949c79b37c92bb3e09e8e/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:27:52.966754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96aff4eb-f84f-4c73-8045-505fae937389 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:52.967238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96aff4eb-f84f-4c73-8045-505fae937389 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:52.967696 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96aff4eb-f84f-4c73-8045-505fae937389 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:52.968173 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f59883-e20f-41ba-8e58-1383e808edd2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:52.969989 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f59883-e20f-41ba-8e58-1383e808edd2 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:52.981309 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1151/2321] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:52 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:52.995901 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-96aff4eb-f84f-4c73-8045-505fae937389 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:52.997655 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-96aff4eb-f84f-4c73-8045-505fae937389 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:53.067874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:53.113643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1171/2322] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:53.124307 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f59883-e20f-41ba-8e58-1383e808edd2 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:53.126438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-26f59883-e20f-41ba-8e58-1383e808edd2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:53.247388 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f59883-e20f-41ba-8e58-1383e808edd2 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:53.393929 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a71002b-954d-4bb2-b4e2-6717e53d028d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:53.394591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a71002b-954d-4bb2-b4e2-6717e53d028d service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:53.394852 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a71002b-954d-4bb2-b4e2-6717e53d028d service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:53.395121 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a71002b-954d-4bb2-b4e2-6717e53d028d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:53.437179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a71002b-954d-4bb2-b4e2-6717e53d028d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:53.438706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a71002b-954d-4bb2-b4e2-6717e53d028d service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:53.462328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1152/2323] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:27:54.805441 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96aff4eb-f84f-4c73-8045-505fae937389 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:54.807084 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-96aff4eb-f84f-4c73-8045-505fae937389 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:54.901353 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96aff4eb-f84f-4c73-8045-505fae937389 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:54.921293 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4331a0ea-6446-46fd-ac13-bcbf287d6e87 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:54.921293 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4331a0ea-6446-46fd-ac13-bcbf287d6e87 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:54.921293 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4331a0ea-6446-46fd-ac13-bcbf287d6e87 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:54.921293 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4331a0ea-6446-46fd-ac13-bcbf287d6e87 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:54.923000 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4331a0ea-6446-46fd-ac13-bcbf287d6e87 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:55.027665 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4331a0ea-6446-46fd-ac13-bcbf287d6e87 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:55.029590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4331a0ea-6446-46fd-ac13-bcbf287d6e87 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:55.050477 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1172/2324] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:54 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:27:55.555304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038163d6-9a33-46cf-94b3-bbe3174a299c None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:55.555861 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038163d6-9a33-46cf-94b3-bbe3174a299c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:55.555861 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038163d6-9a33-46cf-94b3-bbe3174a299c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:55.631099 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-038163d6-9a33-46cf-94b3-bbe3174a299c None None] MFA Rules not processed for user `5d732bd750a54d8cb516fd5756d64936`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:27:55.632167 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-038163d6-9a33-46cf-94b3-bbe3174a299c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:55.695353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:55.761329 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1153/2325] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:27:55 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:27:55.762155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4331a0ea-6446-46fd-ac13-bcbf287d6e87 service cinder] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:55.763839 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4331a0ea-6446-46fd-ac13-bcbf287d6e87 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:27:55.792986 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4331a0ea-6446-46fd-ac13-bcbf287d6e87 service cinder] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:55.813712 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4738d6-bcd5-42e7-9901-0a271946b858 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:55.814362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4738d6-bcd5-42e7-9901-0a271946b858 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:55.814616 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4738d6-bcd5-42e7-9901-0a271946b858 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:55.814891 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4738d6-bcd5-42e7-9901-0a271946b858 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:55.815538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4738d6-bcd5-42e7-9901-0a271946b858 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:55.817964 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4738d6-bcd5-42e7-9901-0a271946b858 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:55.871739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1173/2326] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:27:55 2022] POST /identity/v3/projects => generated 385 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:55.875702 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038163d6-9a33-46cf-94b3-bbe3174a299c None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:55.891578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038163d6-9a33-46cf-94b3-bbe3174a299c None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:55.971143 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4738d6-bcd5-42e7-9901-0a271946b858 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:56.033413 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11984fc0-9e6a-4c29-847d-9b2f86505c5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:56.034628 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11984fc0-9e6a-4c29-847d-9b2f86505c5d admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:56.035098 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11984fc0-9e6a-4c29-847d-9b2f86505c5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:56.035661 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11984fc0-9e6a-4c29-847d-9b2f86505c5d admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:56.036321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11984fc0-9e6a-4c29-847d-9b2f86505c5d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:56.040493 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11984fc0-9e6a-4c29-847d-9b2f86505c5d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:56.076578 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4738d6-bcd5-42e7-9901-0a271946b858 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:56.079045 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1154/2327] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:27:55 2022] POST /identity/v3/users => generated 299 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:27:56.082844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11984fc0-9e6a-4c29-847d-9b2f86505c5d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:56.084453 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18323f14-433c-45d8-8b9f-95e8e93799f0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:56.085230 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18323f14-433c-45d8-8b9f-95e8e93799f0 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:56.085562 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18323f14-433c-45d8-8b9f-95e8e93799f0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:56.085842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18323f14-433c-45d8-8b9f-95e8e93799f0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:56.115189 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11984fc0-9e6a-4c29-847d-9b2f86505c5d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:56.142046 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8c27a7-3f59-4069-b15a-dcdbaad14faa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:56.143561 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8c27a7-3f59-4069-b15a-dcdbaad14faa admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:56.145063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8c27a7-3f59-4069-b15a-dcdbaad14faa admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:56.145676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8c27a7-3f59-4069-b15a-dcdbaad14faa admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:56.148173 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8c27a7-3f59-4069-b15a-dcdbaad14faa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:56.151291 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8c27a7-3f59-4069-b15a-dcdbaad14faa admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:56.180391 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1155/2328] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:56 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:56.184567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8c27a7-3f59-4069-b15a-dcdbaad14faa admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:56.189429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:27:56.202426 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8c27a7-3f59-4069-b15a-dcdbaad14faa admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:56.215843 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7aa669-c5c4-41a0-bde9-24a680137b86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:56.216504 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7aa669-c5c4-41a0-bde9-24a680137b86 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:56.216950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7aa669-c5c4-41a0-bde9-24a680137b86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:56.217332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7aa669-c5c4-41a0-bde9-24a680137b86 admin admin] PATH_INFO: `/v3/projects/c653a70df19945c69e62efc029929b25/users/36ca1ac061ec4658b5c4cc1b436116fb/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:56.239354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18323f14-433c-45d8-8b9f-95e8e93799f0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:56.241191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18323f14-433c-45d8-8b9f-95e8e93799f0 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:56.246428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7aa669-c5c4-41a0-bde9-24a680137b86 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c653a70df19945c69e62efc029929b25, user_id=36ca1ac061ec4658b5c4cc1b436116fb, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:56.249571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7aa669-c5c4-41a0-bde9-24a680137b86 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:56.257449 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1174/2329] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:27:55 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 287 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:27:56.271254 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1156/2330] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:56 2022] PUT /identity/v3/projects/c653a70df19945c69e62efc029929b25/users/36ca1ac061ec4658b5c4cc1b436116fb/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:27:56.275422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18323f14-433c-45d8-8b9f-95e8e93799f0 service cinder] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:56.300211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18323f14-433c-45d8-8b9f-95e8e93799f0 service cinder] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:56.386541 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef205a73-a3bc-46cf-8f55-4f8d84bfe726 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:56.387313 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef205a73-a3bc-46cf-8f55-4f8d84bfe726 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:56.387765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef205a73-a3bc-46cf-8f55-4f8d84bfe726 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:56.387998 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef205a73-a3bc-46cf-8f55-4f8d84bfe726 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:56.388926 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef205a73-a3bc-46cf-8f55-4f8d84bfe726 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:56.390805 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef205a73-a3bc-46cf-8f55-4f8d84bfe726 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:56.426078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1175/2331] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:27:56 2022] GET /identity/v3/roles => generated 1420 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:27:56.427994 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7aa669-c5c4-41a0-bde9-24a680137b86 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:27:56.441091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7aa669-c5c4-41a0-bde9-24a680137b86 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:27:56.455765 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0ec4d2-fe4e-41fb-b88e-cd781ccd20ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:27:56.456771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0ec4d2-fe4e-41fb-b88e-cd781ccd20ac admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:27:56.457083 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0ec4d2-fe4e-41fb-b88e-cd781ccd20ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:27:56.457369 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0ec4d2-fe4e-41fb-b88e-cd781ccd20ac admin admin] PATH_INFO: `/v3/projects/c653a70df19945c69e62efc029929b25/users/36ca1ac061ec4658b5c4cc1b436116fb/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:27:56.459840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d0ec4d2-fe4e-41fb-b88e-cd781ccd20ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=c653a70df19945c69e62efc029929b25, user_id=36ca1ac061ec4658b5c4cc1b436116fb, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:27:56.462612 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d0ec4d2-fe4e-41fb-b88e-cd781ccd20ac admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:27:56.497961 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1157/2332] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:27:56 2022] PUT /identity/v3/projects/c653a70df19945c69e62efc029929b25/users/36ca1ac061ec4658b5c4cc1b436116fb/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:28:00.497287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb5ba47-e68a-4962-a6e3-0886b5cf77da None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:00.497855 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb5ba47-e68a-4962-a6e3-0886b5cf77da None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:00.498085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb5ba47-e68a-4962-a6e3-0886b5cf77da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:00.527243 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-ddb5ba47-e68a-4962-a6e3-0886b5cf77da None None] MFA Rules not processed for user `148a7e11786a4a54909c4a0679407b80`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:28:00.528809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ddb5ba47-e68a-4962-a6e3-0886b5cf77da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:00.665611 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1176/2333] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:28:00 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:28:01.225408 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0ec4d2-fe4e-41fb-b88e-cd781ccd20ac admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:01.398789 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0ec4d2-fe4e-41fb-b88e-cd781ccd20ac admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:01.412440 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1468e92c-041d-469a-b722-e4a9d285ae8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:01.416588 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1468e92c-041d-469a-b722-e4a9d285ae8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:01.417184 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1468e92c-041d-469a-b722-e4a9d285ae8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:01.418362 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1468e92c-041d-469a-b722-e4a9d285ae8a admin admin] PATH_INFO: `/v3/users/5a4ddc10cf85459ca0cc876c22b4d2aa` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:01.419725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1468e92c-041d-469a-b722-e4a9d285ae8a admin admin] RBAC: Authorizing `identity:delete_user(user_id=5a4ddc10cf85459ca0cc876c22b4d2aa)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:01.439024 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1468e92c-041d-469a-b722-e4a9d285ae8a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:01.495680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:28:01.570236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1468e92c-041d-469a-b722-e4a9d285ae8a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5a4ddc10cf85459ca0cc876c22b4d2aa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:01.596041 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1468e92c-041d-469a-b722-e4a9d285ae8a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5a4ddc10cf85459ca0cc876c22b4d2aa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:01.614211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1468e92c-041d-469a-b722-e4a9d285ae8a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5a4ddc10cf85459ca0cc876c22b4d2aa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:01.627525 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1468e92c-041d-469a-b722-e4a9d285ae8a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5a4ddc10cf85459ca0cc876c22b4d2aa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:01.628216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1468e92c-041d-469a-b722-e4a9d285ae8a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5a4ddc10cf85459ca0cc876c22b4d2aa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:01.644193 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1158/2334] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:28:01 2022] DELETE /identity/v3/users/5a4ddc10cf85459ca0cc876c22b4d2aa => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:28:01.647562 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8261ad97-a5ef-4953-9fb9-67581edeb7e5 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:01.648033 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8261ad97-a5ef-4953-9fb9-67581edeb7e5 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:01.648226 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8261ad97-a5ef-4953-9fb9-67581edeb7e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:01.691794 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-8261ad97-a5ef-4953-9fb9-67581edeb7e5 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:28:01.701251 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8261ad97-a5ef-4953-9fb9-67581edeb7e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:01.841725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1177/2335] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:28:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:28:01.858232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1468e92c-041d-469a-b722-e4a9d285ae8a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:01.860566 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1468e92c-041d-469a-b722-e4a9d285ae8a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:01.984247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1468e92c-041d-469a-b722-e4a9d285ae8a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:01.994301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802bef91-9e49-4661-93ff-3404dba27e43 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:01.995305 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802bef91-9e49-4661-93ff-3404dba27e43 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:01.998999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802bef91-9e49-4661-93ff-3404dba27e43 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:01.999588 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802bef91-9e49-4661-93ff-3404dba27e43 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:02.030087 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802bef91-9e49-4661-93ff-3404dba27e43 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:02.032372 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802bef91-9e49-4661-93ff-3404dba27e43 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:02.057374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1159/2336] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:28:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:28:02.411937 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b779a0f-3b51-48f7-b970-5c17a1a08c41 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:02.412539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b779a0f-3b51-48f7-b970-5c17a1a08c41 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:02.413013 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b779a0f-3b51-48f7-b970-5c17a1a08c41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:02.461402 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:28:02.486634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-4b779a0f-3b51-48f7-b970-5c17a1a08c41 None None] MFA Rules not processed for user `b05dcf62adc949c79b37c92bb3e09e8e`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:28:02.490549 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4b779a0f-3b51-48f7-b970-5c17a1a08c41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:02.613824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1178/2337] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:28:02 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:28:02.624452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802bef91-9e49-4661-93ff-3404dba27e43 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:02.627396 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-802bef91-9e49-4661-93ff-3404dba27e43 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:02.813881 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802bef91-9e49-4661-93ff-3404dba27e43 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:02.827515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f892ed-a0cb-489f-966c-1a8c37579e70 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:02.828550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f892ed-a0cb-489f-966c-1a8c37579e70 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:02.829258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f892ed-a0cb-489f-966c-1a8c37579e70 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:02.829724 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f892ed-a0cb-489f-966c-1a8c37579e70 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:02.869140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b779a0f-3b51-48f7-b970-5c17a1a08c41 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:02.870881 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4b779a0f-3b51-48f7-b970-5c17a1a08c41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:02.881353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f892ed-a0cb-489f-966c-1a8c37579e70 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:02.890844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f892ed-a0cb-489f-966c-1a8c37579e70 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:02.908232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b779a0f-3b51-48f7-b970-5c17a1a08c41 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:02.923757 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1160/2338] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:28:02 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:28:02.925392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525d726d-dbca-491c-8563-8c4eba5d8e24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:02.926132 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525d726d-dbca-491c-8563-8c4eba5d8e24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:02.926311 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525d726d-dbca-491c-8563-8c4eba5d8e24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:02.926662 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525d726d-dbca-491c-8563-8c4eba5d8e24 admin admin] PATH_INFO: `/v3/projects/58ddc9e9429740e391a4e5399111514a` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:02.927986 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525d726d-dbca-491c-8563-8c4eba5d8e24 admin admin] RBAC: Authorizing `identity:delete_project(project_id=58ddc9e9429740e391a4e5399111514a)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:02.929981 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525d726d-dbca-491c-8563-8c4eba5d8e24 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:03.064130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-525d726d-dbca-491c-8563-8c4eba5d8e24 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '58ddc9e9429740e391a4e5399111514a', 'request_id': 'req-525d726d-dbca-491c-8563-8c4eba5d8e24'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:03.086866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-525d726d-dbca-491c-8563-8c4eba5d8e24 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '58ddc9e9429740e391a4e5399111514a', 'request_id': 'req-525d726d-dbca-491c-8563-8c4eba5d8e24'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:03.117223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-525d726d-dbca-491c-8563-8c4eba5d8e24 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '58ddc9e9429740e391a4e5399111514a', 'request_id': 'req-525d726d-dbca-491c-8563-8c4eba5d8e24'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:03.149013 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-525d726d-dbca-491c-8563-8c4eba5d8e24 admin admin] The token cache is being invalidate because project 58ddc9e9429740e391a4e5399111514a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:28:03.152790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-525d726d-dbca-491c-8563-8c4eba5d8e24 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:03.157177 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-525d726d-dbca-491c-8563-8c4eba5d8e24 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:03.167667 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1179/2339] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:28:02 2022] DELETE /identity/v3/projects/58ddc9e9429740e391a4e5399111514a => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:28:03.168897 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f892ed-a0cb-489f-966c-1a8c37579e70 service nova] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:03.198550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f892ed-a0cb-489f-966c-1a8c37579e70 service nova] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:03.270434 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:28:03.362088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7df3855-5960-442c-918c-f6906b2f7ec8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:03.364300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7df3855-5960-442c-918c-f6906b2f7ec8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:03.364904 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7df3855-5960-442c-918c-f6906b2f7ec8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:03.365481 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7df3855-5960-442c-918c-f6906b2f7ec8 admin admin] PATH_INFO: `/v3/users/0c7ba20acff948db9ecc494b0ae12a55` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:03.370271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7df3855-5960-442c-918c-f6906b2f7ec8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c7ba20acff948db9ecc494b0ae12a55)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:03.374584 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7df3855-5960-442c-918c-f6906b2f7ec8 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:03.496504 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d7df3855-5960-442c-918c-f6906b2f7ec8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c7ba20acff948db9ecc494b0ae12a55'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:03.541824 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d7df3855-5960-442c-918c-f6906b2f7ec8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c7ba20acff948db9ecc494b0ae12a55'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:03.557900 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d7df3855-5960-442c-918c-f6906b2f7ec8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c7ba20acff948db9ecc494b0ae12a55'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:03.579198 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d7df3855-5960-442c-918c-f6906b2f7ec8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c7ba20acff948db9ecc494b0ae12a55'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:03.580136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d7df3855-5960-442c-918c-f6906b2f7ec8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c7ba20acff948db9ecc494b0ae12a55'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:03.592454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1161/2340] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:28:03 2022] DELETE /identity/v3/users/0c7ba20acff948db9ecc494b0ae12a55 => generated 0 bytes in 424 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:28:04.305287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525d726d-dbca-491c-8563-8c4eba5d8e24 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:04.307391 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-525d726d-dbca-491c-8563-8c4eba5d8e24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:04.350056 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525d726d-dbca-491c-8563-8c4eba5d8e24 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:04.516700 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98617e0-84ca-40a4-99fb-4bdcd137dcd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:04.518651 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98617e0-84ca-40a4-99fb-4bdcd137dcd5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:04.519158 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98617e0-84ca-40a4-99fb-4bdcd137dcd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:04.519771 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98617e0-84ca-40a4-99fb-4bdcd137dcd5 admin admin] PATH_INFO: `/v3/projects/dda000b6a4464d89b8e86f754cac24c2` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:04.522381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98617e0-84ca-40a4-99fb-4bdcd137dcd5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dda000b6a4464d89b8e86f754cac24c2)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:04.527744 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98617e0-84ca-40a4-99fb-4bdcd137dcd5 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:04.587193 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:28:04.619648 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f98617e0-84ca-40a4-99fb-4bdcd137dcd5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dda000b6a4464d89b8e86f754cac24c2', 'request_id': 'req-f98617e0-84ca-40a4-99fb-4bdcd137dcd5'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:04.626100 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f98617e0-84ca-40a4-99fb-4bdcd137dcd5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dda000b6a4464d89b8e86f754cac24c2', 'request_id': 'req-f98617e0-84ca-40a4-99fb-4bdcd137dcd5'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:04.635913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f98617e0-84ca-40a4-99fb-4bdcd137dcd5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dda000b6a4464d89b8e86f754cac24c2', 'request_id': 'req-f98617e0-84ca-40a4-99fb-4bdcd137dcd5'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:04.652772 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f98617e0-84ca-40a4-99fb-4bdcd137dcd5 admin admin] The token cache is being invalidate because project dda000b6a4464d89b8e86f754cac24c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:28:04.653126 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f98617e0-84ca-40a4-99fb-4bdcd137dcd5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:04.653761 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f98617e0-84ca-40a4-99fb-4bdcd137dcd5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:04.658824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1180/2341] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:28:04 2022] DELETE /identity/v3/projects/dda000b6a4464d89b8e86f754cac24c2 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:28:04.660065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7df3855-5960-442c-918c-f6906b2f7ec8 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:04.661749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d7df3855-5960-442c-918c-f6906b2f7ec8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:04.677025 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7df3855-5960-442c-918c-f6906b2f7ec8 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:04.783601 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb7f22c-dce3-486d-b62f-d1b6d8468f79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:04.784261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb7f22c-dce3-486d-b62f-d1b6d8468f79 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:04.784542 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb7f22c-dce3-486d-b62f-d1b6d8468f79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:04.784815 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb7f22c-dce3-486d-b62f-d1b6d8468f79 admin admin] PATH_INFO: `/v3/users/c373ee5e1669473aa111280fb5b595cb` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:04.786048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb7f22c-dce3-486d-b62f-d1b6d8468f79 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c373ee5e1669473aa111280fb5b595cb)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:04.788483 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb7f22c-dce3-486d-b62f-d1b6d8468f79 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:04.858239 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:28:04.867497 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ceb7f22c-dce3-486d-b62f-d1b6d8468f79 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c373ee5e1669473aa111280fb5b595cb'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:04.885376 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ceb7f22c-dce3-486d-b62f-d1b6d8468f79 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c373ee5e1669473aa111280fb5b595cb'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:04.896184 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ceb7f22c-dce3-486d-b62f-d1b6d8468f79 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c373ee5e1669473aa111280fb5b595cb'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:04.912668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ceb7f22c-dce3-486d-b62f-d1b6d8468f79 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c373ee5e1669473aa111280fb5b595cb'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:04.913737 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ceb7f22c-dce3-486d-b62f-d1b6d8468f79 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c373ee5e1669473aa111280fb5b595cb'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:04.923906 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1162/2342] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:28:04 2022] DELETE /identity/v3/users/c373ee5e1669473aa111280fb5b595cb => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:28:05.345648 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e23b6f-6473-48ef-8416-caeeb4171af5 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:05.346191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e23b6f-6473-48ef-8416-caeeb4171af5 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:05.346191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e23b6f-6473-48ef-8416-caeeb4171af5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:05.386860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-43e23b6f-6473-48ef-8416-caeeb4171af5 None None] MFA Rules not processed for user `36ca1ac061ec4658b5c4cc1b436116fb`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:28:05.388101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-43e23b6f-6473-48ef-8416-caeeb4171af5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:05.549968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1181/2343] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:28:05 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:28:05.573762 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb7f22c-dce3-486d-b62f-d1b6d8468f79 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:05.575421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ceb7f22c-dce3-486d-b62f-d1b6d8468f79 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:05.665655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb7f22c-dce3-486d-b62f-d1b6d8468f79 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:05.670771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609ef144-869f-4f6e-93a7-1d2c0e3e67bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:05.671327 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609ef144-869f-4f6e-93a7-1d2c0e3e67bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:05.671683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609ef144-869f-4f6e-93a7-1d2c0e3e67bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:05.672205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609ef144-869f-4f6e-93a7-1d2c0e3e67bc admin admin] PATH_INFO: `/v3/projects/9eae429122fa4946b3f56abcc7e7cfd8` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:05.673821 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609ef144-869f-4f6e-93a7-1d2c0e3e67bc admin admin] RBAC: Authorizing `identity:delete_project(project_id=9eae429122fa4946b3f56abcc7e7cfd8)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:05.676019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609ef144-869f-4f6e-93a7-1d2c0e3e67bc admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:05.738534 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-609ef144-869f-4f6e-93a7-1d2c0e3e67bc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9eae429122fa4946b3f56abcc7e7cfd8', 'request_id': 'req-609ef144-869f-4f6e-93a7-1d2c0e3e67bc'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:05.750827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-609ef144-869f-4f6e-93a7-1d2c0e3e67bc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9eae429122fa4946b3f56abcc7e7cfd8', 'request_id': 'req-609ef144-869f-4f6e-93a7-1d2c0e3e67bc'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:05.759312 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-609ef144-869f-4f6e-93a7-1d2c0e3e67bc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9eae429122fa4946b3f56abcc7e7cfd8', 'request_id': 'req-609ef144-869f-4f6e-93a7-1d2c0e3e67bc'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:05.776616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-609ef144-869f-4f6e-93a7-1d2c0e3e67bc admin admin] The token cache is being invalidate because project 9eae429122fa4946b3f56abcc7e7cfd8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:28:05.776910 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-609ef144-869f-4f6e-93a7-1d2c0e3e67bc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:05.777606 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-609ef144-869f-4f6e-93a7-1d2c0e3e67bc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:05.779406 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1163/2344] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:28:05 2022] DELETE /identity/v3/projects/9eae429122fa4946b3f56abcc7e7cfd8 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:28:05.784027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f861c7ae-410c-483a-a7cb-cd6f2f8ee8c5 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:05.784432 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f861c7ae-410c-483a-a7cb-cd6f2f8ee8c5 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:05.784817 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f861c7ae-410c-483a-a7cb-cd6f2f8ee8c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:05.806723 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-f861c7ae-410c-483a-a7cb-cd6f2f8ee8c5 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:28:05.808037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f861c7ae-410c-483a-a7cb-cd6f2f8ee8c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:05.869448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde39276-f44a-4320-997d-0751b2b387d4 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:05.869782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde39276-f44a-4320-997d-0751b2b387d4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:05.870105 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde39276-f44a-4320-997d-0751b2b387d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:05.892869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-bde39276-f44a-4320-997d-0751b2b387d4 None None] MFA Rules not processed for user `b05dcf62adc949c79b37c92bb3e09e8e`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:28:05.895897 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bde39276-f44a-4320-997d-0751b2b387d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:05.905744 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1182/2345] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:28:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:28:05.915163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73afd822-0f66-4d5c-82cb-680ca125e6c0 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:05.915477 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73afd822-0f66-4d5c-82cb-680ca125e6c0 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:05.916206 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73afd822-0f66-4d5c-82cb-680ca125e6c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:05.939632 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-73afd822-0f66-4d5c-82cb-680ca125e6c0 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:28:05.940982 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-73afd822-0f66-4d5c-82cb-680ca125e6c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:05.957408 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1183/2346] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:28:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:28:05.958447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73afd822-0f66-4d5c-82cb-680ca125e6c0 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:05.971122 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:28:05.973727 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73afd822-0f66-4d5c-82cb-680ca125e6c0 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:05.980045 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1cebdf-941c-4bbc-adbb-442d840aa59b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:05.980752 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1cebdf-941c-4bbc-adbb-442d840aa59b admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:05.981057 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1cebdf-941c-4bbc-adbb-442d840aa59b admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:05.981336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1cebdf-941c-4bbc-adbb-442d840aa59b admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:05.982043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df1cebdf-941c-4bbc-adbb-442d840aa59b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:28:05.982316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df1cebdf-941c-4bbc-adbb-442d840aa59b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:05.983931 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df1cebdf-941c-4bbc-adbb-442d840aa59b admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:05.984580 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1164/2347] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:28:05 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:28:06.005065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1184/2348] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:28:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:28:06.010078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde39276-f44a-4320-997d-0751b2b387d4 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:06.023062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde39276-f44a-4320-997d-0751b2b387d4 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:06.028178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267017d4-ae15-483f-824e-79f14ab98899 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:06.029032 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267017d4-ae15-483f-824e-79f14ab98899 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:06.029359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267017d4-ae15-483f-824e-79f14ab98899 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:06.029638 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267017d4-ae15-483f-824e-79f14ab98899 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:06.030301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267017d4-ae15-483f-824e-79f14ab98899 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:06.032875 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267017d4-ae15-483f-824e-79f14ab98899 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:06.062049 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1165/2349] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:28:06 2022] POST /identity/v3/projects => generated 367 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:28:06.066682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1cebdf-941c-4bbc-adbb-442d840aa59b admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:06.079226 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1cebdf-941c-4bbc-adbb-442d840aa59b admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:06.109501 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:28:06.153689 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab43595-7ba9-4d39-a3ed-5f9d36bc8be9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:06.154651 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab43595-7ba9-4d39-a3ed-5f9d36bc8be9 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:06.154651 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab43595-7ba9-4d39-a3ed-5f9d36bc8be9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:06.154827 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab43595-7ba9-4d39-a3ed-5f9d36bc8be9 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:06.155445 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab43595-7ba9-4d39-a3ed-5f9d36bc8be9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:06.157435 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab43595-7ba9-4d39-a3ed-5f9d36bc8be9 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:06.190673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1185/2350] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:28:06 2022] POST /identity/v3/users => generated 284 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:28:06.193473 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267017d4-ae15-483f-824e-79f14ab98899 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:06.204835 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267017d4-ae15-483f-824e-79f14ab98899 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:06.214930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d828ad92-72bc-4867-83b4-728ae677b759 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:06.215563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d828ad92-72bc-4867-83b4-728ae677b759 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:06.215843 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d828ad92-72bc-4867-83b4-728ae677b759 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:06.216118 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d828ad92-72bc-4867-83b4-728ae677b759 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:06.216775 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d828ad92-72bc-4867-83b4-728ae677b759 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:06.219419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d828ad92-72bc-4867-83b4-728ae677b759 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:06.230151 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1166/2351] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:28:06 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:28:06.234520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab43595-7ba9-4d39-a3ed-5f9d36bc8be9 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:06.247998 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab43595-7ba9-4d39-a3ed-5f9d36bc8be9 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:06.253714 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28208bd9-fb37-4ca6-9c0e-8a5375afa7da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:06.254345 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28208bd9-fb37-4ca6-9c0e-8a5375afa7da admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:06.254630 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28208bd9-fb37-4ca6-9c0e-8a5375afa7da admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:06.254883 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28208bd9-fb37-4ca6-9c0e-8a5375afa7da admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:06.255561 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28208bd9-fb37-4ca6-9c0e-8a5375afa7da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:06.257188 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28208bd9-fb37-4ca6-9c0e-8a5375afa7da admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:06.268866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1186/2352] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:28:06 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:28:06.273268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d828ad92-72bc-4867-83b4-728ae677b759 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:06.283742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d828ad92-72bc-4867-83b4-728ae677b759 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:06.288460 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e93812c-612f-45da-a5cf-45513f9a6eff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:06.289142 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e93812c-612f-45da-a5cf-45513f9a6eff admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:06.289427 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e93812c-612f-45da-a5cf-45513f9a6eff admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:06.289705 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e93812c-612f-45da-a5cf-45513f9a6eff admin admin] PATH_INFO: `/v3/projects/09121f11f90c491f82f0b88b59c0716d/users/e4f125ab5db94c61bc7914175c882049/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:06.304995 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e93812c-612f-45da-a5cf-45513f9a6eff admin admin] RBAC: Authorizing `identity:create_grant(project_id=09121f11f90c491f82f0b88b59c0716d, user_id=e4f125ab5db94c61bc7914175c882049, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:06.307992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e93812c-612f-45da-a5cf-45513f9a6eff admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:06.323498 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1167/2353] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:28:06 2022] PUT /identity/v3/projects/09121f11f90c491f82f0b88b59c0716d/users/e4f125ab5db94c61bc7914175c882049/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:28:06.338367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41850ea-c72f-4eca-bfbf-63b8ed510451 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:06.338724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41850ea-c72f-4eca-bfbf-63b8ed510451 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:06.339050 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41850ea-c72f-4eca-bfbf-63b8ed510451 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:06.370881 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-c41850ea-c72f-4eca-bfbf-63b8ed510451 None None] MFA Rules not processed for user `e4f125ab5db94c61bc7914175c882049`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:28:06.372843 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c41850ea-c72f-4eca-bfbf-63b8ed510451 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:06.442914 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1187/2354] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:28:06 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:28:07.633517 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e93812c-612f-45da-a5cf-45513f9a6eff admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:07.637462 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8e93812c-612f-45da-a5cf-45513f9a6eff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:07.711037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e93812c-612f-45da-a5cf-45513f9a6eff admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:07.717140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fc2787-e71c-42cf-85bd-3e1953ddd451 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '354b8228661c4369b3bca2e631f3f140', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:07.718186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fc2787-e71c-42cf-85bd-3e1953ddd451 service glance] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:07.718946 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fc2787-e71c-42cf-85bd-3e1953ddd451 service glance] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:07.719413 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fc2787-e71c-42cf-85bd-3e1953ddd451 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:07.722632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-93fc2787-e71c-42cf-85bd-3e1953ddd451 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:07.801492 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fc2787-e71c-42cf-85bd-3e1953ddd451 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:07.802954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fc2787-e71c-42cf-85bd-3e1953ddd451 service glance] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:07.811783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1168/2355] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:28:07 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:28:07.902083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41850ea-c72f-4eca-bfbf-63b8ed510451 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:07.911983 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41850ea-c72f-4eca-bfbf-63b8ed510451 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:07.915854 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59e3a1a-e602-4ad8-8b78-d10424d0484f service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '354b8228661c4369b3bca2e631f3f140', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:07.916551 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59e3a1a-e602-4ad8-8b78-d10424d0484f service glance] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:07.916896 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59e3a1a-e602-4ad8-8b78-d10424d0484f service glance] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:07.922807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59e3a1a-e602-4ad8-8b78-d10424d0484f service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:07.931647 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59e3a1a-e602-4ad8-8b78-d10424d0484f service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:07.933276 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59e3a1a-e602-4ad8-8b78-d10424d0484f service glance] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:07.943564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1188/2356] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:28:07 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:28:08.071914 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492bc548-d251-41c6-89f1-c7a544a27165 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:08.073491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492bc548-d251-41c6-89f1-c7a544a27165 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:08.073925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492bc548-d251-41c6-89f1-c7a544a27165 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:08.102534 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-492bc548-d251-41c6-89f1-c7a544a27165 None None] MFA Rules not processed for user `00aeab772672403984bdcb8baaa1520e`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:28:08.104314 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-492bc548-d251-41c6-89f1-c7a544a27165 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:08.163205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:28:08.181208 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1169/2357] 10.222.0.85 () {58 vars in 1009 bytes} [Tue Aug 30 04:28:08 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:28:08.242472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59e3a1a-e602-4ad8-8b78-d10424d0484f service glance] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:08.244483 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e59e3a1a-e602-4ad8-8b78-d10424d0484f service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:08.353860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59e3a1a-e602-4ad8-8b78-d10424d0484f service glance] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:08.362325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4b9323-405b-404f-8e07-3d586445e719 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cb539ef6a69406baa70f660e1a815b4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:08.363015 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4b9323-405b-404f-8e07-3d586445e719 service swift] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:08.363298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4b9323-405b-404f-8e07-3d586445e719 service swift] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:08.363576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4b9323-405b-404f-8e07-3d586445e719 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:08.378997 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4b9323-405b-404f-8e07-3d586445e719 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:08.381151 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4b9323-405b-404f-8e07-3d586445e719 service swift] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:08.405586 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1189/2358] 10.222.0.85 () {60 vars in 1384 bytes} [Tue Aug 30 04:28:08 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 04:28:18.175436 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:28:18.393478 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=94,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:28:21.741812 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bc3455-9492-4118-aad0-6c262be3d9c9 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:21.742355 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bc3455-9492-4118-aad0-6c262be3d9c9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:21.742548 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bc3455-9492-4118-aad0-6c262be3d9c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:21.778454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-62bc3455-9492-4118-aad0-6c262be3d9c9 None None] MFA Rules not processed for user `5d732bd750a54d8cb516fd5756d64936`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:28:21.779679 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-62bc3455-9492-4118-aad0-6c262be3d9c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:21.787569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1170/2359] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:28:21 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:28:31.772616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:28:33.164588 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68530a7c-f08c-4f3e-9562-4ed66cccad79 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:33.165119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68530a7c-f08c-4f3e-9562-4ed66cccad79 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:33.165368 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68530a7c-f08c-4f3e-9562-4ed66cccad79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:33.184636 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-68530a7c-f08c-4f3e-9562-4ed66cccad79 None None] MFA Rules not processed for user `00aeab772672403984bdcb8baaa1520e`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:28:33.185870 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-68530a7c-f08c-4f3e-9562-4ed66cccad79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:33.192969 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1190/2360] 10.222.0.85 () {58 vars in 1009 bytes} [Tue Aug 30 04:28:33 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:28:33.205840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bc3455-9492-4118-aad0-6c262be3d9c9 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:33.217957 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bc3455-9492-4118-aad0-6c262be3d9c9 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:33.224031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77257fe-2971-4fde-8f12-3df836e0702a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cb539ef6a69406baa70f660e1a815b4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:33.224782 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77257fe-2971-4fde-8f12-3df836e0702a service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:33.225225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77257fe-2971-4fde-8f12-3df836e0702a service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:33.225738 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77257fe-2971-4fde-8f12-3df836e0702a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:33.234763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77257fe-2971-4fde-8f12-3df836e0702a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:33.237028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77257fe-2971-4fde-8f12-3df836e0702a service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:33.245963 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1171/2361] 10.222.0.85 () {60 vars in 1384 bytes} [Tue Aug 30 04:28:33 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 04:28:33.490922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68530a7c-f08c-4f3e-9562-4ed66cccad79 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:33.493493 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-68530a7c-f08c-4f3e-9562-4ed66cccad79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:33.588837 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68530a7c-f08c-4f3e-9562-4ed66cccad79 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:33.594101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67fdd302-2058-48ab-8574-79d82adf36cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:33.594858 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fdd302-2058-48ab-8574-79d82adf36cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:33.595245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fdd302-2058-48ab-8574-79d82adf36cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:33.595598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fdd302-2058-48ab-8574-79d82adf36cc admin admin] PATH_INFO: `/v3/users/e99b62187e244a7e8d4144f9357f1499` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:33.598661 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67fdd302-2058-48ab-8574-79d82adf36cc admin admin] RBAC: Authorizing `identity:delete_user(user_id=e99b62187e244a7e8d4144f9357f1499)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:33.600769 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67fdd302-2058-48ab-8574-79d82adf36cc admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:33.669744 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-67fdd302-2058-48ab-8574-79d82adf36cc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e99b62187e244a7e8d4144f9357f1499'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:33.686534 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-67fdd302-2058-48ab-8574-79d82adf36cc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e99b62187e244a7e8d4144f9357f1499'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:33.687410 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-67fdd302-2058-48ab-8574-79d82adf36cc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e99b62187e244a7e8d4144f9357f1499'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:33.708083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-67fdd302-2058-48ab-8574-79d82adf36cc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e99b62187e244a7e8d4144f9357f1499'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:33.709002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-67fdd302-2058-48ab-8574-79d82adf36cc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e99b62187e244a7e8d4144f9357f1499'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:33.735493 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1191/2362] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:28:33 2022] DELETE /identity/v3/users/e99b62187e244a7e8d4144f9357f1499 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:28:33.737500 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f605e87-0ee6-4c8c-a3be-0d1e1769e89f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:33.737893 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f605e87-0ee6-4c8c-a3be-0d1e1769e89f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:33.738274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f605e87-0ee6-4c8c-a3be-0d1e1769e89f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:33.764045 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-1f605e87-0ee6-4c8c-a3be-0d1e1769e89f None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:28:33.768021 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1f605e87-0ee6-4c8c-a3be-0d1e1769e89f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:33.871712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1172/2363] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:28:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:28:33.879505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67fdd302-2058-48ab-8574-79d82adf36cc admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:33.881107 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-67fdd302-2058-48ab-8574-79d82adf36cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:33.998296 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67fdd302-2058-48ab-8574-79d82adf36cc admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:34.026271 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8632eba6-f2a2-41ff-82b2-3e0c323c9af3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:34.027208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8632eba6-f2a2-41ff-82b2-3e0c323c9af3 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:34.027911 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8632eba6-f2a2-41ff-82b2-3e0c323c9af3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:34.028220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8632eba6-f2a2-41ff-82b2-3e0c323c9af3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:34.038897 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8632eba6-f2a2-41ff-82b2-3e0c323c9af3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:34.040403 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8632eba6-f2a2-41ff-82b2-3e0c323c9af3 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:34.051970 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1192/2364] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:28:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:28:34.402168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f605e87-0ee6-4c8c-a3be-0d1e1769e89f None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:28:34.403837 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1f605e87-0ee6-4c8c-a3be-0d1e1769e89f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:28:34.420884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f605e87-0ee6-4c8c-a3be-0d1e1769e89f None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:28:34.428045 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331465cb-e6bc-4230-855b-6638bf4d99f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:28:34.428964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331465cb-e6bc-4230-855b-6638bf4d99f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:28:34.429472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331465cb-e6bc-4230-855b-6638bf4d99f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:28:34.429928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331465cb-e6bc-4230-855b-6638bf4d99f6 admin admin] PATH_INFO: `/v3/projects/74880f6720d34c2f9257465620224459` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:28:34.431754 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331465cb-e6bc-4230-855b-6638bf4d99f6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=74880f6720d34c2f9257465620224459)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:28:34.434211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331465cb-e6bc-4230-855b-6638bf4d99f6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:28:34.493364 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-331465cb-e6bc-4230-855b-6638bf4d99f6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '74880f6720d34c2f9257465620224459', 'request_id': 'req-331465cb-e6bc-4230-855b-6638bf4d99f6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:34.499026 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-331465cb-e6bc-4230-855b-6638bf4d99f6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '74880f6720d34c2f9257465620224459', 'request_id': 'req-331465cb-e6bc-4230-855b-6638bf4d99f6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:34.507621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-331465cb-e6bc-4230-855b-6638bf4d99f6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '74880f6720d34c2f9257465620224459', 'request_id': 'req-331465cb-e6bc-4230-855b-6638bf4d99f6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:34.526237 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-331465cb-e6bc-4230-855b-6638bf4d99f6 admin admin] The token cache is being invalidate because project 74880f6720d34c2f9257465620224459 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:28:34.526622 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-331465cb-e6bc-4230-855b-6638bf4d99f6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:34.527377 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-331465cb-e6bc-4230-855b-6638bf4d99f6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:28:34.529521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1173/2365] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:28:34 2022] DELETE /identity/v3/projects/74880f6720d34c2f9257465620224459 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:28:44.052836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=82,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:28:44.523005 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=60,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:29:13.512906 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8632eba6-f2a2-41ff-82b2-3e0c323c9af3 service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:29:13.514352 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8632eba6-f2a2-41ff-82b2-3e0c323c9af3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:29:13.527026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8632eba6-f2a2-41ff-82b2-3e0c323c9af3 service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:29:13.583867 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46ee914-fa57-4cd7-a2f4-1f811abc9d48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:29:13.584494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46ee914-fa57-4cd7-a2f4-1f811abc9d48 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:29:13.584773 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46ee914-fa57-4cd7-a2f4-1f811abc9d48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:29:13.585062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46ee914-fa57-4cd7-a2f4-1f811abc9d48 admin admin] PATH_INFO: `/v3/users/e4f125ab5db94c61bc7914175c882049` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:29:13.586239 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46ee914-fa57-4cd7-a2f4-1f811abc9d48 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4f125ab5db94c61bc7914175c882049)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:29:13.588154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46ee914-fa57-4cd7-a2f4-1f811abc9d48 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:29:13.639710 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b46ee914-fa57-4cd7-a2f4-1f811abc9d48 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4f125ab5db94c61bc7914175c882049'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:13.650671 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b46ee914-fa57-4cd7-a2f4-1f811abc9d48 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4f125ab5db94c61bc7914175c882049'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:13.651400 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b46ee914-fa57-4cd7-a2f4-1f811abc9d48 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4f125ab5db94c61bc7914175c882049'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:13.668154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b46ee914-fa57-4cd7-a2f4-1f811abc9d48 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4f125ab5db94c61bc7914175c882049'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:13.668871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b46ee914-fa57-4cd7-a2f4-1f811abc9d48 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4f125ab5db94c61bc7914175c882049'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:13.681392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1193/2366] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:29:13 2022] DELETE /identity/v3/users/e4f125ab5db94c61bc7914175c882049 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:29:13.686022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4facd8-68bb-4a22-9b62-8cbbfd2fb594 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:29:13.686562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4facd8-68bb-4a22-9b62-8cbbfd2fb594 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:29:13.686677 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4facd8-68bb-4a22-9b62-8cbbfd2fb594 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:29:13.706435 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-1e4facd8-68bb-4a22-9b62-8cbbfd2fb594 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:29:13.707643 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1e4facd8-68bb-4a22-9b62-8cbbfd2fb594 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:29:13.771247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1174/2367] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:29:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:29:13.781630 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46ee914-fa57-4cd7-a2f4-1f811abc9d48 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:29:13.783085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b46ee914-fa57-4cd7-a2f4-1f811abc9d48 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:29:13.849843 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46ee914-fa57-4cd7-a2f4-1f811abc9d48 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:29:13.854265 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00091ad3-6da3-4740-9f2e-5e22858cd3ec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:29:13.855136 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00091ad3-6da3-4740-9f2e-5e22858cd3ec service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:29:13.855373 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00091ad3-6da3-4740-9f2e-5e22858cd3ec service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:29:13.855656 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00091ad3-6da3-4740-9f2e-5e22858cd3ec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:29:13.869438 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00091ad3-6da3-4740-9f2e-5e22858cd3ec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:29:13.871211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00091ad3-6da3-4740-9f2e-5e22858cd3ec service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:29:13.883911 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1194/2368] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:29:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:29:14.190217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4facd8-68bb-4a22-9b62-8cbbfd2fb594 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:29:14.191718 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1e4facd8-68bb-4a22-9b62-8cbbfd2fb594 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:29:14.203805 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4facd8-68bb-4a22-9b62-8cbbfd2fb594 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:29:14.208978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dea4c37-bbc0-44cc-8de8-f57a4a93d9b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:29:14.209644 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dea4c37-bbc0-44cc-8de8-f57a4a93d9b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:29:14.209928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dea4c37-bbc0-44cc-8de8-f57a4a93d9b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:29:14.210281 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dea4c37-bbc0-44cc-8de8-f57a4a93d9b6 admin admin] PATH_INFO: `/v3/projects/09121f11f90c491f82f0b88b59c0716d` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:29:14.213844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dea4c37-bbc0-44cc-8de8-f57a4a93d9b6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=09121f11f90c491f82f0b88b59c0716d)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:29:14.215900 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dea4c37-bbc0-44cc-8de8-f57a4a93d9b6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:29:14.266834 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1dea4c37-bbc0-44cc-8de8-f57a4a93d9b6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '09121f11f90c491f82f0b88b59c0716d', 'request_id': 'req-1dea4c37-bbc0-44cc-8de8-f57a4a93d9b6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:14.272612 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1dea4c37-bbc0-44cc-8de8-f57a4a93d9b6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '09121f11f90c491f82f0b88b59c0716d', 'request_id': 'req-1dea4c37-bbc0-44cc-8de8-f57a4a93d9b6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:14.280745 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1dea4c37-bbc0-44cc-8de8-f57a4a93d9b6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '09121f11f90c491f82f0b88b59c0716d', 'request_id': 'req-1dea4c37-bbc0-44cc-8de8-f57a4a93d9b6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:14.295282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1dea4c37-bbc0-44cc-8de8-f57a4a93d9b6 admin admin] The token cache is being invalidate because project 09121f11f90c491f82f0b88b59c0716d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:29:14.295576 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1dea4c37-bbc0-44cc-8de8-f57a4a93d9b6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:14.296317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1dea4c37-bbc0-44cc-8de8-f57a4a93d9b6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:14.298398 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1175/2369] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:29:14 2022] DELETE /identity/v3/projects/09121f11f90c491f82f0b88b59c0716d => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:29:18.320892 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00091ad3-6da3-4740-9f2e-5e22858cd3ec service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:29:18.322362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-00091ad3-6da3-4740-9f2e-5e22858cd3ec service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:29:18.349245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:29:18.394956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00091ad3-6da3-4740-9f2e-5e22858cd3ec service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:29:18.457085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c604c4-f98a-4403-8164-f7117931a40e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '359d79cbfcad49beba2b495066824128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:29:18.457820 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c604c4-f98a-4403-8164-f7117931a40e service placement] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:29:18.458127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c604c4-f98a-4403-8164-f7117931a40e service placement] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:29:18.458525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c604c4-f98a-4403-8164-f7117931a40e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:29:18.460440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d7c604c4-f98a-4403-8164-f7117931a40e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:29:18.472069 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c604c4-f98a-4403-8164-f7117931a40e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:29:18.473640 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c604c4-f98a-4403-8164-f7117931a40e service placement] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:29:18.483418 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1195/2370] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:29:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:29:24.291592 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:29:28.479210 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=48 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:29:31.779295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d94780-47d2-48e8-829c-50d59d12e944 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:29:31.779874 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d94780-47d2-48e8-829c-50d59d12e944 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:29:31.780300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d94780-47d2-48e8-829c-50d59d12e944 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:29:31.800433 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b2d94780-47d2-48e8-829c-50d59d12e944 None None] MFA Rules not processed for user `00aeab772672403984bdcb8baaa1520e`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:29:31.801733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b2d94780-47d2-48e8-829c-50d59d12e944 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:29:31.874465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1176/2371] 10.222.0.85 () {58 vars in 1009 bytes} [Tue Aug 30 04:29:31 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:29:31.899376 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c604c4-f98a-4403-8164-f7117931a40e service placement] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:29:31.901159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d7c604c4-f98a-4403-8164-f7117931a40e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:29:31.994250 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c604c4-f98a-4403-8164-f7117931a40e service placement] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:29:31.999712 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0697161d-b490-474a-b27a-78c6e1487251 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cb539ef6a69406baa70f660e1a815b4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:29:32.000407 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0697161d-b490-474a-b27a-78c6e1487251 service swift] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:29:32.000688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0697161d-b490-474a-b27a-78c6e1487251 service swift] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:29:32.000959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0697161d-b490-474a-b27a-78c6e1487251 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:29:32.009110 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0697161d-b490-474a-b27a-78c6e1487251 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:29:32.010782 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0697161d-b490-474a-b27a-78c6e1487251 service swift] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:29:32.018943 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1196/2372] 10.222.0.85 () {60 vars in 1384 bytes} [Tue Aug 30 04:29:31 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 30 04:29:35.895453 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af018033-80d3-4a0f-a37c-2ce16ce86496 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:29:35.895992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af018033-80d3-4a0f-a37c-2ce16ce86496 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:29:35.895992 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af018033-80d3-4a0f-a37c-2ce16ce86496 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:29:35.919930 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-af018033-80d3-4a0f-a37c-2ce16ce86496 None None] MFA Rules not processed for user `cbf574ed57d446bc8fabe7813e64e92c`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:29:35.921870 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-af018033-80d3-4a0f-a37c-2ce16ce86496 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:29:35.993949 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1177/2373] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:29:35 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:29:42.016927 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=32 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:29:45.984487 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=50 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:29:47.628632 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1eada9-e3c2-4fe0-bce0-b00e2eb2e862 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:29:47.629617 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1eada9-e3c2-4fe0-bce0-b00e2eb2e862 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:29:47.630060 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1eada9-e3c2-4fe0-bce0-b00e2eb2e862 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:29:47.669676 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-dd1eada9-e3c2-4fe0-bce0-b00e2eb2e862 None None] MFA Rules not processed for user `3a25d172babb4cd698fac79d7412d25b`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:29:47.670950 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-dd1eada9-e3c2-4fe0-bce0-b00e2eb2e862 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:29:47.766076 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1197/2374] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:29:47 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:29:47.901328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af018033-80d3-4a0f-a37c-2ce16ce86496 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:29:47.902816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-af018033-80d3-4a0f-a37c-2ce16ce86496 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:29:48.019358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af018033-80d3-4a0f-a37c-2ce16ce86496 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:29:48.025322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47006b3-cb68-4f32-8026-ff37644fcda3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:29:48.026051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47006b3-cb68-4f32-8026-ff37644fcda3 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:29:48.026398 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47006b3-cb68-4f32-8026-ff37644fcda3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:29:48.026743 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47006b3-cb68-4f32-8026-ff37644fcda3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:29:48.028627 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b47006b3-cb68-4f32-8026-ff37644fcda3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:29:48.100046 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47006b3-cb68-4f32-8026-ff37644fcda3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:29:48.101702 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47006b3-cb68-4f32-8026-ff37644fcda3 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:29:48.128154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1178/2375] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:29:47 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:29:57.292613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3812dce0-e617-42d0-80b2-450c368cd7ca None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:29:57.293174 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3812dce0-e617-42d0-80b2-450c368cd7ca None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:29:57.293274 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3812dce0-e617-42d0-80b2-450c368cd7ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:29:57.313877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-3812dce0-e617-42d0-80b2-450c368cd7ca None None] MFA Rules not processed for user `00aeab772672403984bdcb8baaa1520e`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:29:57.315114 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-3812dce0-e617-42d0-80b2-450c368cd7ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:29:57.323927 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1198/2376] 10.222.0.85 () {58 vars in 967 bytes} [Tue Aug 30 04:29:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:29:57.337418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47006b3-cb68-4f32-8026-ff37644fcda3 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:29:57.348664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47006b3-cb68-4f32-8026-ff37644fcda3 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:29:57.353856 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d197e99-ba38-447b-913a-ac98487ab29a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cb539ef6a69406baa70f660e1a815b4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:29:57.354471 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d197e99-ba38-447b-913a-ac98487ab29a service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:29:57.354752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d197e99-ba38-447b-913a-ac98487ab29a service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:29:57.355028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d197e99-ba38-447b-913a-ac98487ab29a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:29:57.364294 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d197e99-ba38-447b-913a-ac98487ab29a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:29:57.366149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d197e99-ba38-447b-913a-ac98487ab29a service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:29:57.374800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1179/2377] 10.222.0.85 () {60 vars in 1384 bytes} [Tue Aug 30 04:29:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 04:29:59.479397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3812dce0-e617-42d0-80b2-450c368cd7ca None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:29:59.481401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-3812dce0-e617-42d0-80b2-450c368cd7ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:29:59.555236 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3812dce0-e617-42d0-80b2-450c368cd7ca None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:29:59.559689 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dc1916-0ffd-4c55-8951-37b58376e689 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:29:59.560309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dc1916-0ffd-4c55-8951-37b58376e689 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:29:59.561230 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dc1916-0ffd-4c55-8951-37b58376e689 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:29:59.561512 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dc1916-0ffd-4c55-8951-37b58376e689 admin admin] PATH_INFO: `/v3/users/148a7e11786a4a54909c4a0679407b80` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:29:59.562647 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9dc1916-0ffd-4c55-8951-37b58376e689 admin admin] RBAC: Authorizing `identity:delete_user(user_id=148a7e11786a4a54909c4a0679407b80)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:29:59.564550 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9dc1916-0ffd-4c55-8951-37b58376e689 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:29:59.611918 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e9dc1916-0ffd-4c55-8951-37b58376e689 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '148a7e11786a4a54909c4a0679407b80'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:59.619996 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e9dc1916-0ffd-4c55-8951-37b58376e689 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '148a7e11786a4a54909c4a0679407b80'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:59.620674 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e9dc1916-0ffd-4c55-8951-37b58376e689 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '148a7e11786a4a54909c4a0679407b80'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:59.634185 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e9dc1916-0ffd-4c55-8951-37b58376e689 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '148a7e11786a4a54909c4a0679407b80'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:59.635023 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e9dc1916-0ffd-4c55-8951-37b58376e689 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '148a7e11786a4a54909c4a0679407b80'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:59.648001 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1199/2378] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:29:59 2022] DELETE /identity/v3/users/148a7e11786a4a54909c4a0679407b80 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:29:59.823089 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d197e99-ba38-447b-913a-ac98487ab29a service swift] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:29:59.837166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d197e99-ba38-447b-913a-ac98487ab29a service swift] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:29:59.894921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d3726a-5efd-49d6-bc7d-7ee6ad51a8e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:29:59.895578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d3726a-5efd-49d6-bc7d-7ee6ad51a8e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:29:59.895857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d3726a-5efd-49d6-bc7d-7ee6ad51a8e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:29:59.896131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d3726a-5efd-49d6-bc7d-7ee6ad51a8e5 admin admin] PATH_INFO: `/v3/projects/bdbf61cc87384ade8eb07b032bb6292b` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:29:59.897302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d3726a-5efd-49d6-bc7d-7ee6ad51a8e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bdbf61cc87384ade8eb07b032bb6292b)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:29:59.899242 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d3726a-5efd-49d6-bc7d-7ee6ad51a8e5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:29:59.939454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:29:59.947881 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-79d3726a-5efd-49d6-bc7d-7ee6ad51a8e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bdbf61cc87384ade8eb07b032bb6292b', 'request_id': 'req-79d3726a-5efd-49d6-bc7d-7ee6ad51a8e5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:59.954029 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-79d3726a-5efd-49d6-bc7d-7ee6ad51a8e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bdbf61cc87384ade8eb07b032bb6292b', 'request_id': 'req-79d3726a-5efd-49d6-bc7d-7ee6ad51a8e5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:59.960949 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-79d3726a-5efd-49d6-bc7d-7ee6ad51a8e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bdbf61cc87384ade8eb07b032bb6292b', 'request_id': 'req-79d3726a-5efd-49d6-bc7d-7ee6ad51a8e5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:59.972199 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-79d3726a-5efd-49d6-bc7d-7ee6ad51a8e5 admin admin] The token cache is being invalidate because project bdbf61cc87384ade8eb07b032bb6292b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:29:59.972489 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-79d3726a-5efd-49d6-bc7d-7ee6ad51a8e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:59.973152 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-79d3726a-5efd-49d6-bc7d-7ee6ad51a8e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:29:59.974628 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1180/2379] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:29:59 2022] DELETE /identity/v3/projects/bdbf61cc87384ade8eb07b032bb6292b => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:29:59.980418 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb19d176-10db-4db7-9b80-29b855b11aab None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:29:59.980732 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb19d176-10db-4db7-9b80-29b855b11aab None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:29:59.981005 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb19d176-10db-4db7-9b80-29b855b11aab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:00.000917 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-fb19d176-10db-4db7-9b80-29b855b11aab None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:30:00.002078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-fb19d176-10db-4db7-9b80-29b855b11aab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:00.013934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:30:00.062768 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1200/2380] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:29:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:30:00.070571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3fcade-672a-412c-897d-ef73fc7cc139 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:00.070857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3fcade-672a-412c-897d-ef73fc7cc139 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:00.071135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3fcade-672a-412c-897d-ef73fc7cc139 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:00.090471 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-da3fcade-672a-412c-897d-ef73fc7cc139 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:30:00.091663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-da3fcade-672a-412c-897d-ef73fc7cc139 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:00.099368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1181/2381] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:30:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:30:00.103465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb19d176-10db-4db7-9b80-29b855b11aab None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:00.113590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb19d176-10db-4db7-9b80-29b855b11aab None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:00.118527 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143f7a37-1490-413d-a40b-a137cdad8273 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:00.119191 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143f7a37-1490-413d-a40b-a137cdad8273 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:00.119472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143f7a37-1490-413d-a40b-a137cdad8273 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:00.119746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143f7a37-1490-413d-a40b-a137cdad8273 admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:00.120427 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143f7a37-1490-413d-a40b-a137cdad8273 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:30:00.120761 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143f7a37-1490-413d-a40b-a137cdad8273 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:00.122405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143f7a37-1490-413d-a40b-a137cdad8273 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:00.135598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1201/2382] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:30:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:30:00.139669 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3fcade-672a-412c-897d-ef73fc7cc139 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:00.148951 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3fcade-672a-412c-897d-ef73fc7cc139 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:00.153652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d492e5ab-0631-4c34-a9d0-451fb574fde3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:00.154249 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d492e5ab-0631-4c34-a9d0-451fb574fde3 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:00.154527 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d492e5ab-0631-4c34-a9d0-451fb574fde3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:00.154800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d492e5ab-0631-4c34-a9d0-451fb574fde3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:00.155471 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d492e5ab-0631-4c34-a9d0-451fb574fde3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:00.157405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d492e5ab-0631-4c34-a9d0-451fb574fde3 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:00.185628 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1182/2383] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:30:00 2022] POST /identity/v3/projects => generated 375 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:30:00.190411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143f7a37-1490-413d-a40b-a137cdad8273 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:00.201207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143f7a37-1490-413d-a40b-a137cdad8273 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:00.259771 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71ed78d-4213-4470-b4fa-02dfd2a8c519 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:00.260354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71ed78d-4213-4470-b4fa-02dfd2a8c519 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:00.260668 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71ed78d-4213-4470-b4fa-02dfd2a8c519 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:00.260974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71ed78d-4213-4470-b4fa-02dfd2a8c519 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:00.261699 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71ed78d-4213-4470-b4fa-02dfd2a8c519 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:00.263623 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71ed78d-4213-4470-b4fa-02dfd2a8c519 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:00.296776 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1202/2384] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:30:00 2022] POST /identity/v3/users => generated 288 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:30:00.296885 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d492e5ab-0631-4c34-a9d0-451fb574fde3 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:00.300772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d492e5ab-0631-4c34-a9d0-451fb574fde3 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:00.305362 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008a8d20-a948-4e5e-aa4d-a0fe03e7b29f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:00.305955 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008a8d20-a948-4e5e-aa4d-a0fe03e7b29f admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:00.306236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008a8d20-a948-4e5e-aa4d-a0fe03e7b29f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:00.306511 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008a8d20-a948-4e5e-aa4d-a0fe03e7b29f admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:00.307154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008a8d20-a948-4e5e-aa4d-a0fe03e7b29f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:00.308753 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008a8d20-a948-4e5e-aa4d-a0fe03e7b29f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:00.319081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1183/2385] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:30:00 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:30:00.322790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71ed78d-4213-4470-b4fa-02dfd2a8c519 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:00.332912 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71ed78d-4213-4470-b4fa-02dfd2a8c519 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:00.337479 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6045677-fe87-404c-baa2-3d1ba41dd57d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:00.338066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6045677-fe87-404c-baa2-3d1ba41dd57d admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:00.338339 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6045677-fe87-404c-baa2-3d1ba41dd57d admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:00.338607 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6045677-fe87-404c-baa2-3d1ba41dd57d admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:00.339242 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6045677-fe87-404c-baa2-3d1ba41dd57d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:00.340809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6045677-fe87-404c-baa2-3d1ba41dd57d admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:00.351235 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1203/2386] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:30:00 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:30:00.355269 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008a8d20-a948-4e5e-aa4d-a0fe03e7b29f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:00.365733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008a8d20-a948-4e5e-aa4d-a0fe03e7b29f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:00.370740 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1105a320-484c-4c73-bfc4-2c44889a504d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:00.371381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1105a320-484c-4c73-bfc4-2c44889a504d admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:00.371657 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1105a320-484c-4c73-bfc4-2c44889a504d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:00.371928 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1105a320-484c-4c73-bfc4-2c44889a504d admin admin] PATH_INFO: `/v3/projects/bd0f4805461b4753b83b7659e947bbfa/users/eabd3d2e3ddf411b829578db58eb8ce7/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:00.387319 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1105a320-484c-4c73-bfc4-2c44889a504d admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd0f4805461b4753b83b7659e947bbfa, user_id=eabd3d2e3ddf411b829578db58eb8ce7, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:00.390163 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1105a320-484c-4c73-bfc4-2c44889a504d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:00.403096 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1184/2387] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:30:00 2022] PUT /identity/v3/projects/bd0f4805461b4753b83b7659e947bbfa/users/eabd3d2e3ddf411b829578db58eb8ce7/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:30:00.413779 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df119b6-ffbc-4a29-9411-1f85bd9062de None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:00.414075 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df119b6-ffbc-4a29-9411-1f85bd9062de None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:00.414344 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df119b6-ffbc-4a29-9411-1f85bd9062de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:00.443649 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-5df119b6-ffbc-4a29-9411-1f85bd9062de None None] MFA Rules not processed for user `eabd3d2e3ddf411b829578db58eb8ce7`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:30:00.445248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-5df119b6-ffbc-4a29-9411-1f85bd9062de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:00.511466 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1204/2388] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:30:00 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:30:00.522205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1105a320-484c-4c73-bfc4-2c44889a504d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:00.523960 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1105a320-484c-4c73-bfc4-2c44889a504d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:00.580821 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1105a320-484c-4c73-bfc4-2c44889a504d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:00.584478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83c5a6d-1257-49a8-9189-20f6e884f0ce service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:00.585113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83c5a6d-1257-49a8-9189-20f6e884f0ce service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:00.585401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83c5a6d-1257-49a8-9189-20f6e884f0ce service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:00.585682 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83c5a6d-1257-49a8-9189-20f6e884f0ce service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:00.595084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83c5a6d-1257-49a8-9189-20f6e884f0ce service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:00.596541 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83c5a6d-1257-49a8-9189-20f6e884f0ce service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:00.610140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1185/2389] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:30:00 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:30:10.508477 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=91,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:30:10.604424 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=63,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:30:10.856198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df119b6-ffbc-4a29-9411-1f85bd9062de None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:10.917264 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df119b6-ffbc-4a29-9411-1f85bd9062de None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:10.921784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a5e74c-0263-4ddd-86a8-0372416af106 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:10.922406 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a5e74c-0263-4ddd-86a8-0372416af106 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:10.922688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a5e74c-0263-4ddd-86a8-0372416af106 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:10.922967 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a5e74c-0263-4ddd-86a8-0372416af106 admin admin] PATH_INFO: `/v3/users/eabd3d2e3ddf411b829578db58eb8ce7` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:10.924069 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a5e74c-0263-4ddd-86a8-0372416af106 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eabd3d2e3ddf411b829578db58eb8ce7)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:10.926041 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a5e74c-0263-4ddd-86a8-0372416af106 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:10.973702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-70a5e74c-0263-4ddd-86a8-0372416af106 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eabd3d2e3ddf411b829578db58eb8ce7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:10.981415 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-70a5e74c-0263-4ddd-86a8-0372416af106 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eabd3d2e3ddf411b829578db58eb8ce7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:10.982033 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-70a5e74c-0263-4ddd-86a8-0372416af106 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eabd3d2e3ddf411b829578db58eb8ce7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:10.994279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-70a5e74c-0263-4ddd-86a8-0372416af106 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eabd3d2e3ddf411b829578db58eb8ce7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:10.994892 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-70a5e74c-0263-4ddd-86a8-0372416af106 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eabd3d2e3ddf411b829578db58eb8ce7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:11.007533 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1205/2390] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:30:10 2022] DELETE /identity/v3/users/eabd3d2e3ddf411b829578db58eb8ce7 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:30:11.012151 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be86eb90-f796-4916-bade-a2c57601d1cf None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:11.012430 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be86eb90-f796-4916-bade-a2c57601d1cf None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:11.012701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be86eb90-f796-4916-bade-a2c57601d1cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:11.033199 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-be86eb90-f796-4916-bade-a2c57601d1cf None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:30:11.034415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-be86eb90-f796-4916-bade-a2c57601d1cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:11.094453 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1186/2391] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:30:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:30:11.104218 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a5e74c-0263-4ddd-86a8-0372416af106 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:11.105716 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-70a5e74c-0263-4ddd-86a8-0372416af106 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:11.165661 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a5e74c-0263-4ddd-86a8-0372416af106 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:11.169448 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942a7dad-c0f0-49f6-a93a-8b1935bcaeb2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:11.170056 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942a7dad-c0f0-49f6-a93a-8b1935bcaeb2 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:11.170334 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942a7dad-c0f0-49f6-a93a-8b1935bcaeb2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:11.170605 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942a7dad-c0f0-49f6-a93a-8b1935bcaeb2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:11.179906 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942a7dad-c0f0-49f6-a93a-8b1935bcaeb2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:11.181367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942a7dad-c0f0-49f6-a93a-8b1935bcaeb2 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:11.191429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1206/2392] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:30:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:30:11.536063 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be86eb90-f796-4916-bade-a2c57601d1cf None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:11.537506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-be86eb90-f796-4916-bade-a2c57601d1cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:11.554761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be86eb90-f796-4916-bade-a2c57601d1cf None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:11.559647 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c513cf-2d3f-405e-b245-6d833a3ac79e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:11.560274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c513cf-2d3f-405e-b245-6d833a3ac79e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:11.560622 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c513cf-2d3f-405e-b245-6d833a3ac79e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:11.560847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c513cf-2d3f-405e-b245-6d833a3ac79e admin admin] PATH_INFO: `/v3/projects/bd0f4805461b4753b83b7659e947bbfa` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:11.562088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c513cf-2d3f-405e-b245-6d833a3ac79e admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd0f4805461b4753b83b7659e947bbfa)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:11.564808 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c513cf-2d3f-405e-b245-6d833a3ac79e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:11.614118 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c1c513cf-2d3f-405e-b245-6d833a3ac79e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd0f4805461b4753b83b7659e947bbfa', 'request_id': 'req-c1c513cf-2d3f-405e-b245-6d833a3ac79e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:11.620098 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c1c513cf-2d3f-405e-b245-6d833a3ac79e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd0f4805461b4753b83b7659e947bbfa', 'request_id': 'req-c1c513cf-2d3f-405e-b245-6d833a3ac79e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:11.628961 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c1c513cf-2d3f-405e-b245-6d833a3ac79e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd0f4805461b4753b83b7659e947bbfa', 'request_id': 'req-c1c513cf-2d3f-405e-b245-6d833a3ac79e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:11.640988 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c1c513cf-2d3f-405e-b245-6d833a3ac79e admin admin] The token cache is being invalidate because project bd0f4805461b4753b83b7659e947bbfa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:30:11.641296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c1c513cf-2d3f-405e-b245-6d833a3ac79e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:11.641945 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c1c513cf-2d3f-405e-b245-6d833a3ac79e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:11.643464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1187/2393] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:30:11 2022] DELETE /identity/v3/projects/bd0f4805461b4753b83b7659e947bbfa => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:30:11.648456 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168c06ea-9f3b-4ecb-a0d4-9b60314dc0f6 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:11.648909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168c06ea-9f3b-4ecb-a0d4-9b60314dc0f6 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:11.649060 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168c06ea-9f3b-4ecb-a0d4-9b60314dc0f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:11.668785 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-168c06ea-9f3b-4ecb-a0d4-9b60314dc0f6 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:30:11.670001 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-168c06ea-9f3b-4ecb-a0d4-9b60314dc0f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:11.687037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:30:11.729966 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1207/2394] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:30:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:30:11.738292 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67286350-87dd-4ab9-9ab0-8dab8735c839 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:11.738580 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67286350-87dd-4ab9-9ab0-8dab8735c839 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:11.738857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67286350-87dd-4ab9-9ab0-8dab8735c839 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:11.758437 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-67286350-87dd-4ab9-9ab0-8dab8735c839 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:30:11.759643 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-67286350-87dd-4ab9-9ab0-8dab8735c839 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:11.768030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1188/2395] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:30:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:30:11.772339 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168c06ea-9f3b-4ecb-a0d4-9b60314dc0f6 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:11.782446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168c06ea-9f3b-4ecb-a0d4-9b60314dc0f6 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:11.786980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec6b796-0580-4749-ab38-d9379374c74d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:11.787570 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec6b796-0580-4749-ab38-d9379374c74d admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:11.788034 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec6b796-0580-4749-ab38-d9379374c74d admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:11.788251 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec6b796-0580-4749-ab38-d9379374c74d admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:11.788894 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec6b796-0580-4749-ab38-d9379374c74d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:30:11.789235 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec6b796-0580-4749-ab38-d9379374c74d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:11.790871 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec6b796-0580-4749-ab38-d9379374c74d admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:11.803331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1208/2396] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:30:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:30:11.807747 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67286350-87dd-4ab9-9ab0-8dab8735c839 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:11.816913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67286350-87dd-4ab9-9ab0-8dab8735c839 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:11.821642 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b5bfa1-068f-4689-96ae-ca1fbe7071e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:11.822243 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b5bfa1-068f-4689-96ae-ca1fbe7071e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:11.822526 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b5bfa1-068f-4689-96ae-ca1fbe7071e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:11.822802 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b5bfa1-068f-4689-96ae-ca1fbe7071e2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:11.823434 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b5bfa1-068f-4689-96ae-ca1fbe7071e2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:11.825415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b5bfa1-068f-4689-96ae-ca1fbe7071e2 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:11.852033 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1189/2397] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:30:11 2022] POST /identity/v3/projects => generated 399 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:30:11.856279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec6b796-0580-4749-ab38-d9379374c74d admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:11.867339 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec6b796-0580-4749-ab38-d9379374c74d admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:11.923612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513615c1-470b-4ad7-bbf6-f247d13a4151 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:11.924208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513615c1-470b-4ad7-bbf6-f247d13a4151 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:11.924489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513615c1-470b-4ad7-bbf6-f247d13a4151 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:11.924758 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513615c1-470b-4ad7-bbf6-f247d13a4151 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:11.925438 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-513615c1-470b-4ad7-bbf6-f247d13a4151 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:11.927394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-513615c1-470b-4ad7-bbf6-f247d13a4151 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:11.947689 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1209/2398] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:30:11 2022] POST /identity/v3/users => generated 300 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:30:11.951864 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b5bfa1-068f-4689-96ae-ca1fbe7071e2 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:11.961912 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b5bfa1-068f-4689-96ae-ca1fbe7071e2 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:11.966478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998e1c9a-9ea7-495d-9020-c0049c81f5a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:11.967070 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998e1c9a-9ea7-495d-9020-c0049c81f5a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:11.967347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998e1c9a-9ea7-495d-9020-c0049c81f5a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:11.967617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998e1c9a-9ea7-495d-9020-c0049c81f5a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:11.968259 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998e1c9a-9ea7-495d-9020-c0049c81f5a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:11.970377 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998e1c9a-9ea7-495d-9020-c0049c81f5a2 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:11.980071 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1190/2399] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:30:11 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:30:11.983789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513615c1-470b-4ad7-bbf6-f247d13a4151 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:11.994339 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513615c1-470b-4ad7-bbf6-f247d13a4151 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:11.998847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8534f03e-933a-40f4-b2c6-2e8682cb7510 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:11.999428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8534f03e-933a-40f4-b2c6-2e8682cb7510 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:11.999705 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8534f03e-933a-40f4-b2c6-2e8682cb7510 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:11.999972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8534f03e-933a-40f4-b2c6-2e8682cb7510 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:12.000624 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8534f03e-933a-40f4-b2c6-2e8682cb7510 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:12.002227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8534f03e-933a-40f4-b2c6-2e8682cb7510 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:12.012678 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1210/2400] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:30:11 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:30:12.016993 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998e1c9a-9ea7-495d-9020-c0049c81f5a2 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:12.027025 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998e1c9a-9ea7-495d-9020-c0049c81f5a2 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:12.032225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0af1d6-288a-4261-814f-9ac43fbaccf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:12.033059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0af1d6-288a-4261-814f-9ac43fbaccf5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:12.033451 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0af1d6-288a-4261-814f-9ac43fbaccf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:12.033837 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0af1d6-288a-4261-814f-9ac43fbaccf5 admin admin] PATH_INFO: `/v3/projects/da3c176487134a52996ac76de1e68ca6/users/58e8ca86dbb642478cae46b5ea1b6866/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:12.048915 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0af1d6-288a-4261-814f-9ac43fbaccf5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=da3c176487134a52996ac76de1e68ca6, user_id=58e8ca86dbb642478cae46b5ea1b6866, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:12.051601 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0af1d6-288a-4261-814f-9ac43fbaccf5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:12.064560 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1191/2401] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:30:12 2022] PUT /identity/v3/projects/da3c176487134a52996ac76de1e68ca6/users/58e8ca86dbb642478cae46b5ea1b6866/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:30:12.074451 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438642af-b7f8-4eda-be6e-dce41f0da62f None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:12.074735 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438642af-b7f8-4eda-be6e-dce41f0da62f None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:12.075009 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438642af-b7f8-4eda-be6e-dce41f0da62f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:12.102246 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-438642af-b7f8-4eda-be6e-dce41f0da62f None None] MFA Rules not processed for user `58e8ca86dbb642478cae46b5ea1b6866`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:30:12.103450 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-438642af-b7f8-4eda-be6e-dce41f0da62f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:12.169621 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1211/2402] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:30:12 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:30:12.181418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0af1d6-288a-4261-814f-9ac43fbaccf5 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:12.183094 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2e0af1d6-288a-4261-814f-9ac43fbaccf5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:12.205622 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:30:12.242323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0af1d6-288a-4261-814f-9ac43fbaccf5 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:12.245913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59745224-d58a-4dc1-bf00-a52b0c4ff8a7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:12.246536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59745224-d58a-4dc1-bf00-a52b0c4ff8a7 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:12.246811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59745224-d58a-4dc1-bf00-a52b0c4ff8a7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:12.247079 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59745224-d58a-4dc1-bf00-a52b0c4ff8a7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:12.255893 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59745224-d58a-4dc1-bf00-a52b0c4ff8a7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:12.257338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59745224-d58a-4dc1-bf00-a52b0c4ff8a7 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:12.266201 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1192/2403] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:30:12 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:30:22.166725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:30:22.262265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=17 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:30:26.612720 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438642af-b7f8-4eda-be6e-dce41f0da62f None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:26.680610 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438642af-b7f8-4eda-be6e-dce41f0da62f None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:26.685471 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc80682-e365-487b-a2ec-f4dcf8794d75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:26.686129 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc80682-e365-487b-a2ec-f4dcf8794d75 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:26.686435 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc80682-e365-487b-a2ec-f4dcf8794d75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:26.686730 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc80682-e365-487b-a2ec-f4dcf8794d75 admin admin] PATH_INFO: `/v3/users/58e8ca86dbb642478cae46b5ea1b6866` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:26.687910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bc80682-e365-487b-a2ec-f4dcf8794d75 admin admin] RBAC: Authorizing `identity:delete_user(user_id=58e8ca86dbb642478cae46b5ea1b6866)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:26.690358 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bc80682-e365-487b-a2ec-f4dcf8794d75 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:26.738492 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-9bc80682-e365-487b-a2ec-f4dcf8794d75 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '58e8ca86dbb642478cae46b5ea1b6866'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:26.745995 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-9bc80682-e365-487b-a2ec-f4dcf8794d75 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '58e8ca86dbb642478cae46b5ea1b6866'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:26.746602 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-9bc80682-e365-487b-a2ec-f4dcf8794d75 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '58e8ca86dbb642478cae46b5ea1b6866'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:26.759178 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-9bc80682-e365-487b-a2ec-f4dcf8794d75 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '58e8ca86dbb642478cae46b5ea1b6866'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:26.759975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-9bc80682-e365-487b-a2ec-f4dcf8794d75 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '58e8ca86dbb642478cae46b5ea1b6866'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:26.772601 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1212/2404] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:30:26 2022] DELETE /identity/v3/users/58e8ca86dbb642478cae46b5ea1b6866 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:30:26.777293 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db34bd44-0b9e-403c-aaf6-208fc6fdfefc None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:26.777754 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db34bd44-0b9e-403c-aaf6-208fc6fdfefc None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:26.777866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db34bd44-0b9e-403c-aaf6-208fc6fdfefc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:26.796890 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-db34bd44-0b9e-403c-aaf6-208fc6fdfefc None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:30:26.798121 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-db34bd44-0b9e-403c-aaf6-208fc6fdfefc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:26.855902 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1193/2405] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:30:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:30:26.866759 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc80682-e365-487b-a2ec-f4dcf8794d75 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:26.868153 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-9bc80682-e365-487b-a2ec-f4dcf8794d75 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:26.924791 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc80682-e365-487b-a2ec-f4dcf8794d75 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:26.928464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e0f973-ff3b-45d1-af64-daebd3896a41 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:26.929088 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e0f973-ff3b-45d1-af64-daebd3896a41 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:26.929374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e0f973-ff3b-45d1-af64-daebd3896a41 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:26.929646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e0f973-ff3b-45d1-af64-daebd3896a41 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:26.938517 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e0f973-ff3b-45d1-af64-daebd3896a41 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:26.939954 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e0f973-ff3b-45d1-af64-daebd3896a41 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:26.949345 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1213/2406] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:30:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:30:27.217773 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db34bd44-0b9e-403c-aaf6-208fc6fdfefc None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:27.219216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-db34bd44-0b9e-403c-aaf6-208fc6fdfefc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:27.236374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db34bd44-0b9e-403c-aaf6-208fc6fdfefc None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:27.241353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ad25fd-cf1b-43dd-923c-20ab9bf80985 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:27.241982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ad25fd-cf1b-43dd-923c-20ab9bf80985 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:27.242258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ad25fd-cf1b-43dd-923c-20ab9bf80985 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:27.242528 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ad25fd-cf1b-43dd-923c-20ab9bf80985 admin admin] PATH_INFO: `/v3/projects/da3c176487134a52996ac76de1e68ca6` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:27.243626 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ad25fd-cf1b-43dd-923c-20ab9bf80985 admin admin] RBAC: Authorizing `identity:delete_project(project_id=da3c176487134a52996ac76de1e68ca6)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:27.245555 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ad25fd-cf1b-43dd-923c-20ab9bf80985 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:27.293631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-04ad25fd-cf1b-43dd-923c-20ab9bf80985 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'da3c176487134a52996ac76de1e68ca6', 'request_id': 'req-04ad25fd-cf1b-43dd-923c-20ab9bf80985'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:27.299078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-04ad25fd-cf1b-43dd-923c-20ab9bf80985 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'da3c176487134a52996ac76de1e68ca6', 'request_id': 'req-04ad25fd-cf1b-43dd-923c-20ab9bf80985'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:27.306350 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-04ad25fd-cf1b-43dd-923c-20ab9bf80985 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'da3c176487134a52996ac76de1e68ca6', 'request_id': 'req-04ad25fd-cf1b-43dd-923c-20ab9bf80985'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:27.318130 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-04ad25fd-cf1b-43dd-923c-20ab9bf80985 admin admin] The token cache is being invalidate because project da3c176487134a52996ac76de1e68ca6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:30:27.318425 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-04ad25fd-cf1b-43dd-923c-20ab9bf80985 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:27.319068 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-04ad25fd-cf1b-43dd-923c-20ab9bf80985 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:30:27.320556 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1194/2407] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:30:27 2022] DELETE /identity/v3/projects/da3c176487134a52996ac76de1e68ca6 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:30:36.390607 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff51c9c-70d9-45f9-b58f-302078f13206 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:36.391104 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff51c9c-70d9-45f9-b58f-302078f13206 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:36.391201 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff51c9c-70d9-45f9-b58f-302078f13206 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:36.412162 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-dff51c9c-70d9-45f9-b58f-302078f13206 None None] MFA Rules not processed for user `00aeab772672403984bdcb8baaa1520e`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:30:36.413390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-dff51c9c-70d9-45f9-b58f-302078f13206 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:36.430405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:30:36.469602 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1214/2408] 10.222.0.85 () {58 vars in 1009 bytes} [Tue Aug 30 04:30:36 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:30:36.491983 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ad25fd-cf1b-43dd-923c-20ab9bf80985 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:36.493414 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-04ad25fd-cf1b-43dd-923c-20ab9bf80985 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:36.561901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ad25fd-cf1b-43dd-923c-20ab9bf80985 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:36.566786 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d1e45c-2a84-43ac-a473-864353275741 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cb539ef6a69406baa70f660e1a815b4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:36.567399 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d1e45c-2a84-43ac-a473-864353275741 service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:36.567679 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d1e45c-2a84-43ac-a473-864353275741 service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:36.567948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d1e45c-2a84-43ac-a473-864353275741 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:36.576098 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d1e45c-2a84-43ac-a473-864353275741 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:36.577743 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d1e45c-2a84-43ac-a473-864353275741 service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:36.585694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1195/2409] 10.222.0.85 () {60 vars in 1384 bytes} [Tue Aug 30 04:30:36 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 04:30:46.463156 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=15 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:30:46.582744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=86,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:30:56.045493 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2836e2e6-7108-46c4-ae8f-97cc5c687345 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:56.045998 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2836e2e6-7108-46c4-ae8f-97cc5c687345 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:56.046090 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2836e2e6-7108-46c4-ae8f-97cc5c687345 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:56.047523 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1215/2410] 10.222.0.85 () {54 vars in 921 bytes} [Tue Aug 30 04:30:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:30:56.054117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee63e5fc-5407-4cd7-abd9-e14a8ef4b9f8 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:56.054564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee63e5fc-5407-4cd7-abd9-e14a8ef4b9f8 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:56.054742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee63e5fc-5407-4cd7-abd9-e14a8ef4b9f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:56.075388 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ee63e5fc-5407-4cd7-abd9-e14a8ef4b9f8 None None] MFA Rules not processed for user `4bc4d455ebf041d488b1199ca58358a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:30:56.076712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ee63e5fc-5407-4cd7-abd9-e14a8ef4b9f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:56.153206 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1196/2411] 10.222.0.85 () {58 vars in 1018 bytes} [Tue Aug 30 04:30:56 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:30:56.164252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2836e2e6-7108-46c4-ae8f-97cc5c687345 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:56.166039 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-2836e2e6-7108-46c4-ae8f-97cc5c687345 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:56.225391 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2836e2e6-7108-46c4-ae8f-97cc5c687345 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:56.228974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95473264-a072-4e0d-89e2-50bfdff04dc9 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '354b8228661c4369b3bca2e631f3f140', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:56.229666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95473264-a072-4e0d-89e2-50bfdff04dc9 service glance] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:56.229947 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95473264-a072-4e0d-89e2-50bfdff04dc9 service glance] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:56.230219 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95473264-a072-4e0d-89e2-50bfdff04dc9 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:56.241082 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95473264-a072-4e0d-89e2-50bfdff04dc9 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:56.242465 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95473264-a072-4e0d-89e2-50bfdff04dc9 service glance] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:56.252963 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1216/2412] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:30:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:30:56.321402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8b7272-0c2e-4d3a-83a5-4dfe31ddb4dc None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:56.321693 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8b7272-0c2e-4d3a-83a5-4dfe31ddb4dc None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:56.321971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8b7272-0c2e-4d3a-83a5-4dfe31ddb4dc None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:56.323352 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1197/2413] 10.222.0.85 () {54 vars in 921 bytes} [Tue Aug 30 04:30:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:30:56.327840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1cf389-6247-4477-bc3d-c57982b8c0ea None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:56.328206 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1cf389-6247-4477-bc3d-c57982b8c0ea None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:56.328532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1cf389-6247-4477-bc3d-c57982b8c0ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:56.347879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-cf1cf389-6247-4477-bc3d-c57982b8c0ea None None] MFA Rules not processed for user `5c2edd77896a4534888d6758ec0ee0bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:30:56.349129 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-cf1cf389-6247-4477-bc3d-c57982b8c0ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:56.399489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1217/2414] 10.222.0.85 () {58 vars in 1018 bytes} [Tue Aug 30 04:30:56 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:30:56.411472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8b7272-0c2e-4d3a-83a5-4dfe31ddb4dc None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:56.413223 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4c8b7272-0c2e-4d3a-83a5-4dfe31ddb4dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:56.426680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8b7272-0c2e-4d3a-83a5-4dfe31ddb4dc None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:56.430179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053322c9-5f98-4743-8703-012e226e2756 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:56.430772 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053322c9-5f98-4743-8703-012e226e2756 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:56.431054 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053322c9-5f98-4743-8703-012e226e2756 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:56.431335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053322c9-5f98-4743-8703-012e226e2756 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:56.438219 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053322c9-5f98-4743-8703-012e226e2756 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:56.439649 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053322c9-5f98-4743-8703-012e226e2756 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:56.447883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1198/2415] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:30:56 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:30:56.530295 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e959f6-84bb-4b4e-8b96-e342bb11d888 None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:56.530584 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e959f6-84bb-4b4e-8b96-e342bb11d888 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:56.530862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e959f6-84bb-4b4e-8b96-e342bb11d888 None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:56.533070 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1218/2416] 10.222.0.85 () {54 vars in 921 bytes} [Tue Aug 30 04:30:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:30:56.544428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba217ba-e66b-4ef3-8bd2-cd53e63d78c6 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:56.544744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba217ba-e66b-4ef3-8bd2-cd53e63d78c6 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:56.545023 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba217ba-e66b-4ef3-8bd2-cd53e63d78c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:56.568332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-8ba217ba-e66b-4ef3-8bd2-cd53e63d78c6 None None] MFA Rules not processed for user `4bc4d455ebf041d488b1199ca58358a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:30:56.569575 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8ba217ba-e66b-4ef3-8bd2-cd53e63d78c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:30:56.584368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1199/2417] 10.222.0.85 () {58 vars in 1018 bytes} [Tue Aug 30 04:30:56 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:30:56.596000 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e959f6-84bb-4b4e-8b96-e342bb11d888 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:30:56.607807 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e959f6-84bb-4b4e-8b96-e342bb11d888 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:30:56.611393 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3489520-464f-49d4-a5eb-502eaa0cd99a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:30:56.611981 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3489520-464f-49d4-a5eb-502eaa0cd99a service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:30:56.612272 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3489520-464f-49d4-a5eb-502eaa0cd99a service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:30:56.612528 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3489520-464f-49d4-a5eb-502eaa0cd99a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:30:56.623353 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3489520-464f-49d4-a5eb-502eaa0cd99a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:30:56.624773 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3489520-464f-49d4-a5eb-502eaa0cd99a service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:30:56.635294 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1219/2418] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:30:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:31:01.743034 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e70176-0556-44ba-bb26-7c592764dd69 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:01.743502 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e70176-0556-44ba-bb26-7c592764dd69 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:01.743651 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e70176-0556-44ba-bb26-7c592764dd69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:01.763194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-29e70176-0556-44ba-bb26-7c592764dd69 None None] MFA Rules not processed for user `00aeab772672403984bdcb8baaa1520e`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:31:01.764396 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-29e70176-0556-44ba-bb26-7c592764dd69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:01.772568 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1200/2419] 10.222.0.85 () {58 vars in 1009 bytes} [Tue Aug 30 04:31:01 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:31:01.785598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3489520-464f-49d4-a5eb-502eaa0cd99a service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:31:01.796796 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3489520-464f-49d4-a5eb-502eaa0cd99a service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:31:01.801907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363fc9d9-8640-4507-acf9-a49db0a77923 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cb539ef6a69406baa70f660e1a815b4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:31:01.802496 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363fc9d9-8640-4507-acf9-a49db0a77923 service swift] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:01.802768 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363fc9d9-8640-4507-acf9-a49db0a77923 service swift] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:01.803033 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363fc9d9-8640-4507-acf9-a49db0a77923 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:01.811496 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363fc9d9-8640-4507-acf9-a49db0a77923 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:31:01.813095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363fc9d9-8640-4507-acf9-a49db0a77923 service swift] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:31:01.821535 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1220/2420] 10.222.0.85 () {60 vars in 1384 bytes} [Tue Aug 30 04:31:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 04:31:09.249226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e70176-0556-44ba-bb26-7c592764dd69 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:31:09.250745 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-29e70176-0556-44ba-bb26-7c592764dd69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:09.336497 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e70176-0556-44ba-bb26-7c592764dd69 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:31:09.341319 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ccf9f8-2804-4eb8-b58f-cb7c24e021e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:31:09.341982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ccf9f8-2804-4eb8-b58f-cb7c24e021e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:09.342260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ccf9f8-2804-4eb8-b58f-cb7c24e021e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:09.342556 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ccf9f8-2804-4eb8-b58f-cb7c24e021e3 admin admin] PATH_INFO: `/v3/users/3a25d172babb4cd698fac79d7412d25b` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:09.343681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ccf9f8-2804-4eb8-b58f-cb7c24e021e3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3a25d172babb4cd698fac79d7412d25b)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:31:09.345734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ccf9f8-2804-4eb8-b58f-cb7c24e021e3 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:31:09.405101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b9ccf9f8-2804-4eb8-b58f-cb7c24e021e3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3a25d172babb4cd698fac79d7412d25b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:09.419999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b9ccf9f8-2804-4eb8-b58f-cb7c24e021e3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3a25d172babb4cd698fac79d7412d25b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:09.422017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:31:09.428622 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b9ccf9f8-2804-4eb8-b58f-cb7c24e021e3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3a25d172babb4cd698fac79d7412d25b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:09.435846 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b9ccf9f8-2804-4eb8-b58f-cb7c24e021e3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3a25d172babb4cd698fac79d7412d25b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:09.436467 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b9ccf9f8-2804-4eb8-b58f-cb7c24e021e3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3a25d172babb4cd698fac79d7412d25b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:09.443801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1201/2421] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:31:09 2022] DELETE /identity/v3/users/3a25d172babb4cd698fac79d7412d25b => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:31:09.630206 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363fc9d9-8640-4507-acf9-a49db0a77923 service swift] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:31:09.641912 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363fc9d9-8640-4507-acf9-a49db0a77923 service swift] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:31:09.712068 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39166924-ab70-44e9-b299-20167ddbce74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:31:09.712617 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39166924-ab70-44e9-b299-20167ddbce74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:09.712893 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39166924-ab70-44e9-b299-20167ddbce74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:09.713223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39166924-ab70-44e9-b299-20167ddbce74 admin admin] PATH_INFO: `/v3/projects/2d8842fc1da94bf681eeb42464032b8a` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:09.714324 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39166924-ab70-44e9-b299-20167ddbce74 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2d8842fc1da94bf681eeb42464032b8a)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:31:09.716217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39166924-ab70-44e9-b299-20167ddbce74 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:31:09.750441 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:31:09.767361 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-39166924-ab70-44e9-b299-20167ddbce74 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2d8842fc1da94bf681eeb42464032b8a', 'request_id': 'req-39166924-ab70-44e9-b299-20167ddbce74'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:09.773292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-39166924-ab70-44e9-b299-20167ddbce74 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2d8842fc1da94bf681eeb42464032b8a', 'request_id': 'req-39166924-ab70-44e9-b299-20167ddbce74'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:09.780868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-39166924-ab70-44e9-b299-20167ddbce74 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2d8842fc1da94bf681eeb42464032b8a', 'request_id': 'req-39166924-ab70-44e9-b299-20167ddbce74'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:09.792977 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-39166924-ab70-44e9-b299-20167ddbce74 admin admin] The token cache is being invalidate because project 2d8842fc1da94bf681eeb42464032b8a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:31:09.793329 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-39166924-ab70-44e9-b299-20167ddbce74 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:09.793969 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-39166924-ab70-44e9-b299-20167ddbce74 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:09.795467 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1221/2422] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:31:09 2022] DELETE /identity/v3/projects/2d8842fc1da94bf681eeb42464032b8a => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:31:09.800852 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f923c2-dddf-483e-90f7-27f915cfa5b9 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:09.801168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f923c2-dddf-483e-90f7-27f915cfa5b9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:09.801446 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f923c2-dddf-483e-90f7-27f915cfa5b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:09.831721 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-53f923c2-dddf-483e-90f7-27f915cfa5b9 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:31:09.834480 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-53f923c2-dddf-483e-90f7-27f915cfa5b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:09.910920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1202/2423] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:31:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:31:09.918754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7532330-56bb-4e7c-b185-41bc31f543fb None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:09.919011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7532330-56bb-4e7c-b185-41bc31f543fb None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:09.919284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7532330-56bb-4e7c-b185-41bc31f543fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:09.941479 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-f7532330-56bb-4e7c-b185-41bc31f543fb None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:31:09.942877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f7532330-56bb-4e7c-b185-41bc31f543fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:09.952749 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1222/2424] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:31:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:31:09.956894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f923c2-dddf-483e-90f7-27f915cfa5b9 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:31:09.968007 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f923c2-dddf-483e-90f7-27f915cfa5b9 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:31:09.972797 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7646f871-ceed-491e-85ad-1b1f955c381c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:31:09.973450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7646f871-ceed-491e-85ad-1b1f955c381c admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:09.973735 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7646f871-ceed-491e-85ad-1b1f955c381c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:09.974009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7646f871-ceed-491e-85ad-1b1f955c381c admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:09.974665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7646f871-ceed-491e-85ad-1b1f955c381c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:31:09.974968 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7646f871-ceed-491e-85ad-1b1f955c381c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:31:09.976761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7646f871-ceed-491e-85ad-1b1f955c381c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:31:09.993472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1203/2425] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:31:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:31:09.997610 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7532330-56bb-4e7c-b185-41bc31f543fb None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:31:10.007991 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7532330-56bb-4e7c-b185-41bc31f543fb None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:31:10.013086 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838dd075-0f12-4e8c-8a4a-ad3fc9927150 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:31:10.013702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838dd075-0f12-4e8c-8a4a-ad3fc9927150 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:10.013986 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838dd075-0f12-4e8c-8a4a-ad3fc9927150 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:10.014262 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838dd075-0f12-4e8c-8a4a-ad3fc9927150 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:10.014908 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838dd075-0f12-4e8c-8a4a-ad3fc9927150 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:31:10.016969 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838dd075-0f12-4e8c-8a4a-ad3fc9927150 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:31:10.044591 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1223/2426] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:31:09 2022] POST /identity/v3/projects => generated 391 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:31:10.048942 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7646f871-ceed-491e-85ad-1b1f955c381c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:31:10.059621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7646f871-ceed-491e-85ad-1b1f955c381c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:31:10.120644 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaba5ef9-18f1-411c-9460-dcf1056c7584 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:31:10.121313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaba5ef9-18f1-411c-9460-dcf1056c7584 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:10.121598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaba5ef9-18f1-411c-9460-dcf1056c7584 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:10.121874 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaba5ef9-18f1-411c-9460-dcf1056c7584 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:10.122511 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaba5ef9-18f1-411c-9460-dcf1056c7584 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:31:10.124443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaba5ef9-18f1-411c-9460-dcf1056c7584 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:31:10.148978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1204/2427] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:31:10 2022] POST /identity/v3/users => generated 296 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:31:10.158014 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838dd075-0f12-4e8c-8a4a-ad3fc9927150 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:31:10.170506 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838dd075-0f12-4e8c-8a4a-ad3fc9927150 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:31:10.175914 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e476cf-3bd1-4d3d-88da-ff67620acc0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:31:10.176542 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e476cf-3bd1-4d3d-88da-ff67620acc0b admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:10.176823 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e476cf-3bd1-4d3d-88da-ff67620acc0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:10.177119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e476cf-3bd1-4d3d-88da-ff67620acc0b admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:10.177767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e476cf-3bd1-4d3d-88da-ff67620acc0b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:31:10.179366 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e476cf-3bd1-4d3d-88da-ff67620acc0b admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:31:10.189576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1224/2428] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:31:10 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:31:10.193608 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaba5ef9-18f1-411c-9460-dcf1056c7584 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:31:10.203457 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaba5ef9-18f1-411c-9460-dcf1056c7584 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:31:10.208304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60935dcf-cf72-4892-b322-2aea5c805c1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:31:10.209000 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60935dcf-cf72-4892-b322-2aea5c805c1e admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:10.209320 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60935dcf-cf72-4892-b322-2aea5c805c1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:10.209599 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60935dcf-cf72-4892-b322-2aea5c805c1e admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:10.210254 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60935dcf-cf72-4892-b322-2aea5c805c1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:31:10.212120 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60935dcf-cf72-4892-b322-2aea5c805c1e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:31:10.224601 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1205/2429] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:31:10 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:31:10.228876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e476cf-3bd1-4d3d-88da-ff67620acc0b admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:31:10.242645 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e476cf-3bd1-4d3d-88da-ff67620acc0b admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:31:10.248242 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972da6fd-4b04-4b24-9198-244fc7afe951 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:31:10.248931 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972da6fd-4b04-4b24-9198-244fc7afe951 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:10.249238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972da6fd-4b04-4b24-9198-244fc7afe951 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:10.249514 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972da6fd-4b04-4b24-9198-244fc7afe951 admin admin] PATH_INFO: `/v3/projects/180fc88701814284ba4c968783accbf1/users/7e36d8a480a34d18acce6a1d7b199ae2/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:10.265328 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972da6fd-4b04-4b24-9198-244fc7afe951 admin admin] RBAC: Authorizing `identity:create_grant(project_id=180fc88701814284ba4c968783accbf1, user_id=7e36d8a480a34d18acce6a1d7b199ae2, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:31:10.267993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972da6fd-4b04-4b24-9198-244fc7afe951 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:31:10.280811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1225/2430] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:31:10 2022] PUT /identity/v3/projects/180fc88701814284ba4c968783accbf1/users/7e36d8a480a34d18acce6a1d7b199ae2/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:31:10.291606 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284baa49-a7d8-4616-b77c-dc98043c2475 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:10.292125 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284baa49-a7d8-4616-b77c-dc98043c2475 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:10.292283 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284baa49-a7d8-4616-b77c-dc98043c2475 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:10.322519 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-284baa49-a7d8-4616-b77c-dc98043c2475 None None] MFA Rules not processed for user `7e36d8a480a34d18acce6a1d7b199ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:31:10.323730 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-284baa49-a7d8-4616-b77c-dc98043c2475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:10.364234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:31:10.391209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1206/2431] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:31:10 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:31:10.402145 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972da6fd-4b04-4b24-9198-244fc7afe951 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:31:10.403819 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-972da6fd-4b04-4b24-9198-244fc7afe951 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:10.470950 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972da6fd-4b04-4b24-9198-244fc7afe951 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:31:10.475437 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb6d301-8454-4c41-afda-a83f22698def service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:31:10.476023 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb6d301-8454-4c41-afda-a83f22698def service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:10.476303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb6d301-8454-4c41-afda-a83f22698def service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:10.476621 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb6d301-8454-4c41-afda-a83f22698def service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:10.485177 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb6d301-8454-4c41-afda-a83f22698def service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:31:10.486606 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb6d301-8454-4c41-afda-a83f22698def service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:31:10.495932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1226/2432] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:31:10 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:31:14.013461 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284baa49-a7d8-4616-b77c-dc98043c2475 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:31:14.015046 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-284baa49-a7d8-4616-b77c-dc98043c2475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:14.071123 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284baa49-a7d8-4616-b77c-dc98043c2475 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:31:14.074681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b6185b-024e-469c-8e6b-4dfeae6c2bc7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:31:14.075278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b6185b-024e-469c-8e6b-4dfeae6c2bc7 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:14.075558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b6185b-024e-469c-8e6b-4dfeae6c2bc7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:14.075855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b6185b-024e-469c-8e6b-4dfeae6c2bc7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:14.077607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-97b6185b-024e-469c-8e6b-4dfeae6c2bc7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:14.149847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b6185b-024e-469c-8e6b-4dfeae6c2bc7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:31:14.151283 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b6185b-024e-469c-8e6b-4dfeae6c2bc7 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:31:14.161941 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1207/2433] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:31:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:31:14.594790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb6d301-8454-4c41-afda-a83f22698def service cinder] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:31:14.604627 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb6d301-8454-4c41-afda-a83f22698def service cinder] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:31:14.608230 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f417f0b6-36eb-4914-90c2-394a1a6fb2fc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:31:14.608821 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f417f0b6-36eb-4914-90c2-394a1a6fb2fc service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:14.609148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f417f0b6-36eb-4914-90c2-394a1a6fb2fc service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:14.609426 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f417f0b6-36eb-4914-90c2-394a1a6fb2fc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:14.611232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f417f0b6-36eb-4914-90c2-394a1a6fb2fc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:14.625352 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f417f0b6-36eb-4914-90c2-394a1a6fb2fc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:31:14.626773 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f417f0b6-36eb-4914-90c2-394a1a6fb2fc service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:31:14.637520 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1227/2434] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:31:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:31:22.835085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b6185b-024e-469c-8e6b-4dfeae6c2bc7 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:31:22.836513 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-97b6185b-024e-469c-8e6b-4dfeae6c2bc7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:22.847849 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b6185b-024e-469c-8e6b-4dfeae6c2bc7 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:31:22.851412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fc5867-c161-418a-854c-d60396a9bf9e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:31:22.852012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fc5867-c161-418a-854c-d60396a9bf9e service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:22.852291 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fc5867-c161-418a-854c-d60396a9bf9e service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:22.852566 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fc5867-c161-418a-854c-d60396a9bf9e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:22.854344 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a9fc5867-c161-418a-854c-d60396a9bf9e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:22.916287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fc5867-c161-418a-854c-d60396a9bf9e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:31:22.917773 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fc5867-c161-418a-854c-d60396a9bf9e service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:31:22.927174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1208/2435] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:31:22 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:31:24.632710 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=73,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:31:25.784251 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f417f0b6-36eb-4914-90c2-394a1a6fb2fc service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:31:25.785741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f417f0b6-36eb-4914-90c2-394a1a6fb2fc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:25.800505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f417f0b6-36eb-4914-90c2-394a1a6fb2fc service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:31:25.806887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7a6ecd-eef4-4f47-8ea1-138327b86464 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:31:25.807740 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7a6ecd-eef4-4f47-8ea1-138327b86464 service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:25.808172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7a6ecd-eef4-4f47-8ea1-138327b86464 service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:25.808594 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7a6ecd-eef4-4f47-8ea1-138327b86464 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:25.810911 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-5a7a6ecd-eef4-4f47-8ea1-138327b86464 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:25.882746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7a6ecd-eef4-4f47-8ea1-138327b86464 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:31:25.884968 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7a6ecd-eef4-4f47-8ea1-138327b86464 service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:31:25.897711 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1228/2436] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:31:25 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:31:31.043275 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fc5867-c161-418a-854c-d60396a9bf9e service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:31:31.065205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:31:31.108044 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fc5867-c161-418a-854c-d60396a9bf9e service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:31:31.113397 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1e2009-244f-4e97-ae5b-310903c35b2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:31:31.114335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1e2009-244f-4e97-ae5b-310903c35b2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:31.114615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1e2009-244f-4e97-ae5b-310903c35b2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:31.114884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1e2009-244f-4e97-ae5b-310903c35b2a admin admin] PATH_INFO: `/v3/users/7e36d8a480a34d18acce6a1d7b199ae2` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:31.116017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1e2009-244f-4e97-ae5b-310903c35b2a admin admin] RBAC: Authorizing `identity:delete_user(user_id=7e36d8a480a34d18acce6a1d7b199ae2)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:31:31.117998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1e2009-244f-4e97-ae5b-310903c35b2a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:31:31.167005 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ce1e2009-244f-4e97-ae5b-310903c35b2a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7e36d8a480a34d18acce6a1d7b199ae2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:31.180524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ce1e2009-244f-4e97-ae5b-310903c35b2a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7e36d8a480a34d18acce6a1d7b199ae2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:31.188323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ce1e2009-244f-4e97-ae5b-310903c35b2a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7e36d8a480a34d18acce6a1d7b199ae2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:31.196308 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ce1e2009-244f-4e97-ae5b-310903c35b2a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7e36d8a480a34d18acce6a1d7b199ae2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:31.196918 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ce1e2009-244f-4e97-ae5b-310903c35b2a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7e36d8a480a34d18acce6a1d7b199ae2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:31.203125 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1209/2437] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:31:31 2022] DELETE /identity/v3/users/7e36d8a480a34d18acce6a1d7b199ae2 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:31:31.207359 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d9c5b2-b83b-4739-98af-1fbb9ea841b4 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:31.207865 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d9c5b2-b83b-4739-98af-1fbb9ea841b4 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:31.207973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d9c5b2-b83b-4739-98af-1fbb9ea841b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:31.229387 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-e1d9c5b2-b83b-4739-98af-1fbb9ea841b4 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:31:31.229387 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e1d9c5b2-b83b-4739-98af-1fbb9ea841b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:31.286443 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1229/2438] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:31:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:31:31.296048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1e2009-244f-4e97-ae5b-310903c35b2a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:31:31.297490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ce1e2009-244f-4e97-ae5b-310903c35b2a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:31.354615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1e2009-244f-4e97-ae5b-310903c35b2a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:31:31.358199 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b19a400-edef-49f6-9686-15577818bc6f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:31:31.358803 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b19a400-edef-49f6-9686-15577818bc6f service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:31.359078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b19a400-edef-49f6-9686-15577818bc6f service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:31.359342 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b19a400-edef-49f6-9686-15577818bc6f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:31.368256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b19a400-edef-49f6-9686-15577818bc6f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:31:31.374895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b19a400-edef-49f6-9686-15577818bc6f service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:31:31.386562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1210/2439] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:31:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:31:31.653566 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d9c5b2-b83b-4739-98af-1fbb9ea841b4 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:31:31.655067 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e1d9c5b2-b83b-4739-98af-1fbb9ea841b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:31.668147 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d9c5b2-b83b-4739-98af-1fbb9ea841b4 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:31:31.672908 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959a0230-5f7a-4945-9bfc-4e7ae74d5a49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:31:31.673578 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959a0230-5f7a-4945-9bfc-4e7ae74d5a49 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:31.673857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959a0230-5f7a-4945-9bfc-4e7ae74d5a49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:31.674128 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959a0230-5f7a-4945-9bfc-4e7ae74d5a49 admin admin] PATH_INFO: `/v3/projects/180fc88701814284ba4c968783accbf1` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:31.675294 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959a0230-5f7a-4945-9bfc-4e7ae74d5a49 admin admin] RBAC: Authorizing `identity:delete_project(project_id=180fc88701814284ba4c968783accbf1)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:31:31.677226 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959a0230-5f7a-4945-9bfc-4e7ae74d5a49 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:31:31.725776 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-959a0230-5f7a-4945-9bfc-4e7ae74d5a49 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '180fc88701814284ba4c968783accbf1', 'request_id': 'req-959a0230-5f7a-4945-9bfc-4e7ae74d5a49'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:31.731508 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-959a0230-5f7a-4945-9bfc-4e7ae74d5a49 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '180fc88701814284ba4c968783accbf1', 'request_id': 'req-959a0230-5f7a-4945-9bfc-4e7ae74d5a49'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:31.739281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-959a0230-5f7a-4945-9bfc-4e7ae74d5a49 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '180fc88701814284ba4c968783accbf1', 'request_id': 'req-959a0230-5f7a-4945-9bfc-4e7ae74d5a49'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:31.751200 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-959a0230-5f7a-4945-9bfc-4e7ae74d5a49 admin admin] The token cache is being invalidate because project 180fc88701814284ba4c968783accbf1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:31:31.751508 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-959a0230-5f7a-4945-9bfc-4e7ae74d5a49 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:31.752132 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-959a0230-5f7a-4945-9bfc-4e7ae74d5a49 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:31:31.754139 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1230/2440] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:31:31 2022] DELETE /identity/v3/projects/180fc88701814284ba4c968783accbf1 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:31:34.632983 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b19a400-edef-49f6-9686-15577818bc6f service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:31:34.634538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2b19a400-edef-49f6-9686-15577818bc6f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:34.646471 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b19a400-edef-49f6-9686-15577818bc6f service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:31:34.686503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:31:34.697260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a30b7b-2c00-401f-a13d-6604e6be0871 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:31:34.697861 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a30b7b-2c00-401f-a13d-6604e6be0871 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:34.698135 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a30b7b-2c00-401f-a13d-6604e6be0871 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:34.698397 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a30b7b-2c00-401f-a13d-6604e6be0871 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:34.700114 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-70a30b7b-2c00-401f-a13d-6604e6be0871 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:34.709690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a30b7b-2c00-401f-a13d-6604e6be0871 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:31:34.711131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a30b7b-2c00-401f-a13d-6604e6be0871 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:31:34.722651 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1211/2441] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:31:34 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:31:37.448154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acb081e-4b5c-4409-832d-a1f9b901821f None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:37.448640 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acb081e-4b5c-4409-832d-a1f9b901821f None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:37.448719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acb081e-4b5c-4409-832d-a1f9b901821f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:37.468600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-3acb081e-4b5c-4409-832d-a1f9b901821f None None] MFA Rules not processed for user `00aeab772672403984bdcb8baaa1520e`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:31:37.469976 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-3acb081e-4b5c-4409-832d-a1f9b901821f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:37.488338 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:31:37.530178 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1231/2442] 10.222.0.85 () {58 vars in 967 bytes} [Tue Aug 30 04:31:37 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:31:37.553253 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a30b7b-2c00-401f-a13d-6604e6be0871 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:31:37.554687 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-70a30b7b-2c00-401f-a13d-6604e6be0871 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:31:37.628122 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a30b7b-2c00-401f-a13d-6604e6be0871 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:31:37.639096 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147e7255-8d4a-44f8-b2aa-dd1d13ddceb1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cb539ef6a69406baa70f660e1a815b4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:31:37.639816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147e7255-8d4a-44f8-b2aa-dd1d13ddceb1 service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:31:37.640274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147e7255-8d4a-44f8-b2aa-dd1d13ddceb1 service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:31:37.640551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147e7255-8d4a-44f8-b2aa-dd1d13ddceb1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:31:37.654215 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147e7255-8d4a-44f8-b2aa-dd1d13ddceb1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:31:37.655811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147e7255-8d4a-44f8-b2aa-dd1d13ddceb1 service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:31:37.664507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1212/2443] 10.222.0.85 () {60 vars in 1384 bytes} [Tue Aug 30 04:31:37 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 04:31:47.523491 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=15 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:31:47.661224 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=39 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:32:02.699957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094dfee6-e4e0-4341-b59d-644561378bff None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:02.700486 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094dfee6-e4e0-4341-b59d-644561378bff None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:02.700602 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094dfee6-e4e0-4341-b59d-644561378bff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:02.719666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-094dfee6-e4e0-4341-b59d-644561378bff None None] MFA Rules not processed for user `cbf574ed57d446bc8fabe7813e64e92c`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:32:02.720863 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-094dfee6-e4e0-4341-b59d-644561378bff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:02.774590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1232/2444] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:32:02 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:32:05.660422 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147e7255-8d4a-44f8-b2aa-dd1d13ddceb1 service swift] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:05.662591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-147e7255-8d4a-44f8-b2aa-dd1d13ddceb1 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:05.746672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147e7255-8d4a-44f8-b2aa-dd1d13ddceb1 service swift] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:05.752275 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960b6a6e-6022-4fc5-8868-66060f8feec1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:05.753216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960b6a6e-6022-4fc5-8868-66060f8feec1 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:05.753216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960b6a6e-6022-4fc5-8868-66060f8feec1 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:05.753596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960b6a6e-6022-4fc5-8868-66060f8feec1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:05.755587 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-960b6a6e-6022-4fc5-8868-66060f8feec1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:05.767908 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-960b6a6e-6022-4fc5-8868-66060f8feec1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:05.769558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-960b6a6e-6022-4fc5-8868-66060f8feec1 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:05.783109 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1213/2445] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:32:05 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:32:12.767448 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=25 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:32:13.864623 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094dfee6-e4e0-4341-b59d-644561378bff None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:13.866160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-094dfee6-e4e0-4341-b59d-644561378bff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:13.884599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094dfee6-e4e0-4341-b59d-644561378bff None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:13.890392 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c47baa7-c5fe-45f2-bdfc-dbbeecec0e0a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:13.891011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c47baa7-c5fe-45f2-bdfc-dbbeecec0e0a service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:13.891294 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c47baa7-c5fe-45f2-bdfc-dbbeecec0e0a service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:13.891574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c47baa7-c5fe-45f2-bdfc-dbbeecec0e0a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:13.893601 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4c47baa7-c5fe-45f2-bdfc-dbbeecec0e0a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:13.906637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c47baa7-c5fe-45f2-bdfc-dbbeecec0e0a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:13.908212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c47baa7-c5fe-45f2-bdfc-dbbeecec0e0a service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:13.918583 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1233/2446] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:32:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:32:15.218830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e64f9f-b2ba-4ab2-87e2-5ff961204977 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:15.219357 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e64f9f-b2ba-4ab2-87e2-5ff961204977 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:15.219515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e64f9f-b2ba-4ab2-87e2-5ff961204977 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:15.239956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-03e64f9f-b2ba-4ab2-87e2-5ff961204977 None None] MFA Rules not processed for user `00aeab772672403984bdcb8baaa1520e`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:32:15.241244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-03e64f9f-b2ba-4ab2-87e2-5ff961204977 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:15.248200 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1214/2447] 10.222.0.85 () {58 vars in 967 bytes} [Tue Aug 30 04:32:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:32:15.260201 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c47baa7-c5fe-45f2-bdfc-dbbeecec0e0a service nova] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:15.271650 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c47baa7-c5fe-45f2-bdfc-dbbeecec0e0a service nova] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:15.276605 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9a881e-1fd5-4a95-becb-c61ea955390c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cb539ef6a69406baa70f660e1a815b4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:15.277228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9a881e-1fd5-4a95-becb-c61ea955390c service swift] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:15.277642 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9a881e-1fd5-4a95-becb-c61ea955390c service swift] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:15.277923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9a881e-1fd5-4a95-becb-c61ea955390c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:15.286987 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab9a881e-1fd5-4a95-becb-c61ea955390c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:15.288724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab9a881e-1fd5-4a95-becb-c61ea955390c service swift] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:15.297147 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1234/2448] 10.222.0.85 () {60 vars in 1384 bytes} [Tue Aug 30 04:32:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 30 04:32:19.032371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e64f9f-b2ba-4ab2-87e2-5ff961204977 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:19.034069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-03e64f9f-b2ba-4ab2-87e2-5ff961204977 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:19.108323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e64f9f-b2ba-4ab2-87e2-5ff961204977 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:19.114345 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca29008-d7e9-448d-8ec6-392957229a03 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '359d79cbfcad49beba2b495066824128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:19.114958 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca29008-d7e9-448d-8ec6-392957229a03 service placement] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:19.115239 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca29008-d7e9-448d-8ec6-392957229a03 service placement] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:19.115536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca29008-d7e9-448d-8ec6-392957229a03 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:19.117569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7ca29008-d7e9-448d-8ec6-392957229a03 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:19.129355 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca29008-d7e9-448d-8ec6-392957229a03 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:19.130999 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca29008-d7e9-448d-8ec6-392957229a03 service placement] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:19.140990 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1215/2449] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:32:19 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:32:22.084250 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9a881e-1fd5-4a95-becb-c61ea955390c service swift] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:22.096699 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9a881e-1fd5-4a95-becb-c61ea955390c service swift] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:22.103826 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dced620-4c34-4824-92ba-ee17a83ae258 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '359d79cbfcad49beba2b495066824128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:22.104706 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dced620-4c34-4824-92ba-ee17a83ae258 service placement] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:22.105224 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dced620-4c34-4824-92ba-ee17a83ae258 service placement] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:22.105710 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dced620-4c34-4824-92ba-ee17a83ae258 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:22.107914 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-6dced620-4c34-4824-92ba-ee17a83ae258 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:22.127192 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dced620-4c34-4824-92ba-ee17a83ae258 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:22.134589 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dced620-4c34-4824-92ba-ee17a83ae258 service placement] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:22.143759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca29008-d7e9-448d-8ec6-392957229a03 service placement] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:22.147150 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7ca29008-d7e9-448d-8ec6-392957229a03 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:22.158144 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1235/2450] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:32:22 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:32:22.220538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca29008-d7e9-448d-8ec6-392957229a03 service placement] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:22.226260 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9e5176-3e1e-4d90-aee3-691151055369 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:22.226993 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9e5176-3e1e-4d90-aee3-691151055369 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:22.227283 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9e5176-3e1e-4d90-aee3-691151055369 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:22.227565 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9e5176-3e1e-4d90-aee3-691151055369 admin admin] PATH_INFO: `/v3/users/5d732bd750a54d8cb516fd5756d64936` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:22.228699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9e5176-3e1e-4d90-aee3-691151055369 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5d732bd750a54d8cb516fd5756d64936)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:22.230684 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9e5176-3e1e-4d90-aee3-691151055369 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:22.241610 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:32:22.280505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ce9e5176-3e1e-4d90-aee3-691151055369 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5d732bd750a54d8cb516fd5756d64936'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:22.293570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ce9e5176-3e1e-4d90-aee3-691151055369 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5d732bd750a54d8cb516fd5756d64936'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:22.302049 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ce9e5176-3e1e-4d90-aee3-691151055369 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5d732bd750a54d8cb516fd5756d64936'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:22.310314 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ce9e5176-3e1e-4d90-aee3-691151055369 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5d732bd750a54d8cb516fd5756d64936'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:22.310942 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ce9e5176-3e1e-4d90-aee3-691151055369 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5d732bd750a54d8cb516fd5756d64936'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:22.321927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1216/2451] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:32:22 2022] DELETE /identity/v3/users/5d732bd750a54d8cb516fd5756d64936 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:32:22.492127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dced620-4c34-4824-92ba-ee17a83ae258 service placement] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:22.503240 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dced620-4c34-4824-92ba-ee17a83ae258 service placement] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:22.560935 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb273db1-5d06-472c-9674-8dd44fadca2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:22.561622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb273db1-5d06-472c-9674-8dd44fadca2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:22.561907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb273db1-5d06-472c-9674-8dd44fadca2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:22.562186 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb273db1-5d06-472c-9674-8dd44fadca2b admin admin] PATH_INFO: `/v3/projects/3a706ffd11144f46ba54becd86464f02` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:22.563405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb273db1-5d06-472c-9674-8dd44fadca2b admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a706ffd11144f46ba54becd86464f02)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:22.565559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb273db1-5d06-472c-9674-8dd44fadca2b admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:22.610602 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bb273db1-5d06-472c-9674-8dd44fadca2b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a706ffd11144f46ba54becd86464f02', 'request_id': 'req-bb273db1-5d06-472c-9674-8dd44fadca2b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:22.616704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bb273db1-5d06-472c-9674-8dd44fadca2b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a706ffd11144f46ba54becd86464f02', 'request_id': 'req-bb273db1-5d06-472c-9674-8dd44fadca2b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:22.623588 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bb273db1-5d06-472c-9674-8dd44fadca2b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a706ffd11144f46ba54becd86464f02', 'request_id': 'req-bb273db1-5d06-472c-9674-8dd44fadca2b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:22.635422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bb273db1-5d06-472c-9674-8dd44fadca2b admin admin] The token cache is being invalidate because project 3a706ffd11144f46ba54becd86464f02 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:32:22.635689 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bb273db1-5d06-472c-9674-8dd44fadca2b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:22.636315 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bb273db1-5d06-472c-9674-8dd44fadca2b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:22.637849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1236/2452] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:32:22 2022] DELETE /identity/v3/projects/3a706ffd11144f46ba54becd86464f02 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:32:22.642008 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9e5176-3e1e-4d90-aee3-691151055369 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:22.643563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ce9e5176-3e1e-4d90-aee3-691151055369 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:22.656490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9e5176-3e1e-4d90-aee3-691151055369 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:22.712121 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc8b4b2-ac20-4b0c-a99d-0bd6cf0203a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:22.712746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc8b4b2-ac20-4b0c-a99d-0bd6cf0203a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:22.713029 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc8b4b2-ac20-4b0c-a99d-0bd6cf0203a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:22.713336 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc8b4b2-ac20-4b0c-a99d-0bd6cf0203a1 admin admin] PATH_INFO: `/v3/users/36ca1ac061ec4658b5c4cc1b436116fb` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:22.714453 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc8b4b2-ac20-4b0c-a99d-0bd6cf0203a1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=36ca1ac061ec4658b5c4cc1b436116fb)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:22.716389 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc8b4b2-ac20-4b0c-a99d-0bd6cf0203a1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:22.763885 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7cc8b4b2-ac20-4b0c-a99d-0bd6cf0203a1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '36ca1ac061ec4658b5c4cc1b436116fb'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:22.775556 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7cc8b4b2-ac20-4b0c-a99d-0bd6cf0203a1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '36ca1ac061ec4658b5c4cc1b436116fb'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:22.782670 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7cc8b4b2-ac20-4b0c-a99d-0bd6cf0203a1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '36ca1ac061ec4658b5c4cc1b436116fb'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:22.790457 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7cc8b4b2-ac20-4b0c-a99d-0bd6cf0203a1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '36ca1ac061ec4658b5c4cc1b436116fb'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:22.791088 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7cc8b4b2-ac20-4b0c-a99d-0bd6cf0203a1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '36ca1ac061ec4658b5c4cc1b436116fb'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:22.797001 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1217/2453] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:32:22 2022] DELETE /identity/v3/users/36ca1ac061ec4658b5c4cc1b436116fb => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:32:22.971503 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb273db1-5d06-472c-9674-8dd44fadca2b admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:22.973620 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-bb273db1-5d06-472c-9674-8dd44fadca2b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:22.986611 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb273db1-5d06-472c-9674-8dd44fadca2b admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:23.037158 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:32:23.047834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adba6655-b232-418c-8020-d88ed4c97747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:23.048468 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adba6655-b232-418c-8020-d88ed4c97747 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:23.048730 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adba6655-b232-418c-8020-d88ed4c97747 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:23.049000 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adba6655-b232-418c-8020-d88ed4c97747 admin admin] PATH_INFO: `/v3/projects/c653a70df19945c69e62efc029929b25` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:23.050117 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adba6655-b232-418c-8020-d88ed4c97747 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c653a70df19945c69e62efc029929b25)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:23.052043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adba6655-b232-418c-8020-d88ed4c97747 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:23.097226 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-adba6655-b232-418c-8020-d88ed4c97747 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c653a70df19945c69e62efc029929b25', 'request_id': 'req-adba6655-b232-418c-8020-d88ed4c97747'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:23.103352 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-adba6655-b232-418c-8020-d88ed4c97747 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c653a70df19945c69e62efc029929b25', 'request_id': 'req-adba6655-b232-418c-8020-d88ed4c97747'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:23.112459 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-adba6655-b232-418c-8020-d88ed4c97747 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c653a70df19945c69e62efc029929b25', 'request_id': 'req-adba6655-b232-418c-8020-d88ed4c97747'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:23.124057 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-adba6655-b232-418c-8020-d88ed4c97747 admin admin] The token cache is being invalidate because project c653a70df19945c69e62efc029929b25 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:32:23.124350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-adba6655-b232-418c-8020-d88ed4c97747 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:23.125055 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-adba6655-b232-418c-8020-d88ed4c97747 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:32:23.126581 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1237/2454] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:32:22 2022] DELETE /identity/v3/projects/c653a70df19945c69e62efc029929b25 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:32:23.131600 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dc42bc-84b7-4ca6-b60f-91a7ea19a165 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:23.131885 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dc42bc-84b7-4ca6-b60f-91a7ea19a165 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:23.132161 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dc42bc-84b7-4ca6-b60f-91a7ea19a165 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:23.146286 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:DELETE=20,keystone:SELECT=78,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:32:23.153914 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-e3dc42bc-84b7-4ca6-b60f-91a7ea19a165 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:32:23.155723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e3dc42bc-84b7-4ca6-b60f-91a7ea19a165 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:23.217405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1218/2455] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:32:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:32:23.281819 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a1be7e-1480-4211-a2ce-ed46e80602aa None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:23.282107 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a1be7e-1480-4211-a2ce-ed46e80602aa None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:23.282385 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a1be7e-1480-4211-a2ce-ed46e80602aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:23.303032 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-42a1be7e-1480-4211-a2ce-ed46e80602aa None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:32:23.304239 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-42a1be7e-1480-4211-a2ce-ed46e80602aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:23.312326 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1238/2456] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:32:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:32:23.316353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dc42bc-84b7-4ca6-b60f-91a7ea19a165 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:23.326158 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dc42bc-84b7-4ca6-b60f-91a7ea19a165 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:23.330658 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004d7726-488d-4115-8e02-23b3ece5d2e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:23.331257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004d7726-488d-4115-8e02-23b3ece5d2e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:23.331534 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004d7726-488d-4115-8e02-23b3ece5d2e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:23.331804 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004d7726-488d-4115-8e02-23b3ece5d2e4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:23.332951 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004d7726-488d-4115-8e02-23b3ece5d2e4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:32:23.333285 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004d7726-488d-4115-8e02-23b3ece5d2e4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:23.334923 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004d7726-488d-4115-8e02-23b3ece5d2e4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:23.346389 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1219/2457] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:32:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:32:23.351026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a1be7e-1480-4211-a2ce-ed46e80602aa None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:23.360230 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a1be7e-1480-4211-a2ce-ed46e80602aa None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:23.365416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2e3f45-7e1b-49ab-ae11-9e55b0a1cb59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:23.366119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2e3f45-7e1b-49ab-ae11-9e55b0a1cb59 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:23.366506 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2e3f45-7e1b-49ab-ae11-9e55b0a1cb59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:23.366882 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2e3f45-7e1b-49ab-ae11-9e55b0a1cb59 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:23.367597 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2e3f45-7e1b-49ab-ae11-9e55b0a1cb59 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:23.369528 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2e3f45-7e1b-49ab-ae11-9e55b0a1cb59 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:23.397777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1239/2458] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:32:23 2022] POST /identity/v3/projects => generated 373 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:32:23.401950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004d7726-488d-4115-8e02-23b3ece5d2e4 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:23.412509 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004d7726-488d-4115-8e02-23b3ece5d2e4 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:23.467745 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157cd15c-8f1d-4c38-9ee5-0efe561498eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:23.468353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157cd15c-8f1d-4c38-9ee5-0efe561498eb admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:23.468633 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157cd15c-8f1d-4c38-9ee5-0efe561498eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:23.468905 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157cd15c-8f1d-4c38-9ee5-0efe561498eb admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:23.469584 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157cd15c-8f1d-4c38-9ee5-0efe561498eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:23.471494 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157cd15c-8f1d-4c38-9ee5-0efe561498eb admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:23.493968 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1220/2459] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:32:23 2022] POST /identity/v3/users => generated 287 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:32:23.498252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2e3f45-7e1b-49ab-ae11-9e55b0a1cb59 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:23.509377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2e3f45-7e1b-49ab-ae11-9e55b0a1cb59 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:23.514883 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f7eef1-4a65-4d24-b96f-dd19b25ed8d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:23.515511 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f7eef1-4a65-4d24-b96f-dd19b25ed8d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:23.515791 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f7eef1-4a65-4d24-b96f-dd19b25ed8d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:23.516068 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f7eef1-4a65-4d24-b96f-dd19b25ed8d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:23.516709 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f7eef1-4a65-4d24-b96f-dd19b25ed8d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:23.518369 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f7eef1-4a65-4d24-b96f-dd19b25ed8d2 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:23.528987 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1240/2460] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:32:23 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:32:23.532962 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157cd15c-8f1d-4c38-9ee5-0efe561498eb admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:23.543021 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157cd15c-8f1d-4c38-9ee5-0efe561498eb admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:23.547679 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc56c4e-c415-4afe-84e4-ee729c34bb29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:23.548273 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc56c4e-c415-4afe-84e4-ee729c34bb29 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:23.548555 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc56c4e-c415-4afe-84e4-ee729c34bb29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:23.548833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc56c4e-c415-4afe-84e4-ee729c34bb29 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:23.549518 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc56c4e-c415-4afe-84e4-ee729c34bb29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:23.551136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc56c4e-c415-4afe-84e4-ee729c34bb29 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:23.564359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1221/2461] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:32:23 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:32:23.565585 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f7eef1-4a65-4d24-b96f-dd19b25ed8d2 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:23.575879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f7eef1-4a65-4d24-b96f-dd19b25ed8d2 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:23.580727 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832bd8d6-2523-45a8-98d5-b59f1ef6a641 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:23.581407 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832bd8d6-2523-45a8-98d5-b59f1ef6a641 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:23.581691 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832bd8d6-2523-45a8-98d5-b59f1ef6a641 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:23.581964 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832bd8d6-2523-45a8-98d5-b59f1ef6a641 admin admin] PATH_INFO: `/v3/projects/96907629895d451697f448cf2c3bc6a1/users/a71c0e86c4b2456abce0e459705a2919/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:23.600025 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832bd8d6-2523-45a8-98d5-b59f1ef6a641 admin admin] RBAC: Authorizing `identity:create_grant(project_id=96907629895d451697f448cf2c3bc6a1, user_id=a71c0e86c4b2456abce0e459705a2919, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:23.602833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832bd8d6-2523-45a8-98d5-b59f1ef6a641 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:23.616362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1241/2462] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:32:23 2022] PUT /identity/v3/projects/96907629895d451697f448cf2c3bc6a1/users/a71c0e86c4b2456abce0e459705a2919/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:32:23.621491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09e7dcb-cc3a-49c8-8e9c-563e5456a252 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:23.621846 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09e7dcb-cc3a-49c8-8e9c-563e5456a252 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:23.622191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09e7dcb-cc3a-49c8-8e9c-563e5456a252 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:23.643064 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c09e7dcb-cc3a-49c8-8e9c-563e5456a252 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:32:23.644289 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c09e7dcb-cc3a-49c8-8e9c-563e5456a252 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:23.707147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1222/2463] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:32:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:32:23.717507 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832bd8d6-2523-45a8-98d5-b59f1ef6a641 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:23.719425 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-832bd8d6-2523-45a8-98d5-b59f1ef6a641 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:23.777328 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832bd8d6-2523-45a8-98d5-b59f1ef6a641 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:23.782035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cfca2b-83b2-4436-a4e1-f77934e839c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:23.782652 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cfca2b-83b2-4436-a4e1-f77934e839c4 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:23.782931 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cfca2b-83b2-4436-a4e1-f77934e839c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:23.783201 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cfca2b-83b2-4436-a4e1-f77934e839c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:23.792808 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41cfca2b-83b2-4436-a4e1-f77934e839c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:23.794282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41cfca2b-83b2-4436-a4e1-f77934e839c4 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:23.804420 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1242/2464] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:32:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:32:27.774864 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea80d33f-15b7-4140-a07c-7ca6e302fc34 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:27.779158 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea80d33f-15b7-4140-a07c-7ca6e302fc34 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:27.779862 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea80d33f-15b7-4140-a07c-7ca6e302fc34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:27.824146 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:32:27.828061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ea80d33f-15b7-4140-a07c-7ca6e302fc34 None None] MFA Rules not processed for user `a71c0e86c4b2456abce0e459705a2919`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:32:27.831816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ea80d33f-15b7-4140-a07c-7ca6e302fc34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:27.930242 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1223/2465] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:32:27 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:32:27.934434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cfca2b-83b2-4436-a4e1-f77934e839c4 service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:27.945351 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cfca2b-83b2-4436-a4e1-f77934e839c4 service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:27.950187 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928fbee5-064d-45a6-9d0b-d64db7ec5153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:27.950797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928fbee5-064d-45a6-9d0b-d64db7ec5153 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:27.951075 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928fbee5-064d-45a6-9d0b-d64db7ec5153 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:27.951346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928fbee5-064d-45a6-9d0b-d64db7ec5153 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:27.951988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928fbee5-064d-45a6-9d0b-d64db7ec5153 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:27.954183 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928fbee5-064d-45a6-9d0b-d64db7ec5153 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:27.978762 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:32:27.994021 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1243/2466] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:32:27 2022] POST /identity/v3/projects => generated 373 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:32:27.994924 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea80d33f-15b7-4140-a07c-7ca6e302fc34 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:28.007787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea80d33f-15b7-4140-a07c-7ca6e302fc34 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:28.075942 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf431ec-ab98-4a24-9b45-81b396d51134 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:28.077909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf431ec-ab98-4a24-9b45-81b396d51134 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:28.078142 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf431ec-ab98-4a24-9b45-81b396d51134 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:28.078671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf431ec-ab98-4a24-9b45-81b396d51134 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:28.079885 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf431ec-ab98-4a24-9b45-81b396d51134 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:28.082081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf431ec-ab98-4a24-9b45-81b396d51134 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:28.110546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1224/2467] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:32:27 2022] POST /identity/v3/users => generated 293 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:32:28.112120 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928fbee5-064d-45a6-9d0b-d64db7ec5153 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:28.124205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928fbee5-064d-45a6-9d0b-d64db7ec5153 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:28.130544 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84325702-9c83-4e5a-b5be-8a4643f8cd5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:28.131285 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84325702-9c83-4e5a-b5be-8a4643f8cd5f admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:28.131590 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84325702-9c83-4e5a-b5be-8a4643f8cd5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:28.131931 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84325702-9c83-4e5a-b5be-8a4643f8cd5f admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:28.132709 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84325702-9c83-4e5a-b5be-8a4643f8cd5f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:28.134495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84325702-9c83-4e5a-b5be-8a4643f8cd5f admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:28.146676 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1244/2468] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:32:28 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:32:28.151997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf431ec-ab98-4a24-9b45-81b396d51134 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:28.165661 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf431ec-ab98-4a24-9b45-81b396d51134 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:28.173423 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a05453-af3a-48cb-a245-3afda0c4fc7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:28.174559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a05453-af3a-48cb-a245-3afda0c4fc7f admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:28.175226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a05453-af3a-48cb-a245-3afda0c4fc7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:28.175900 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a05453-af3a-48cb-a245-3afda0c4fc7f admin admin] PATH_INFO: `/v3/projects/1b05404af98c48dd9c26aafd234846a9/users/1f965e6aca9341edbe6ecd391666993d/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:28.195283 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a05453-af3a-48cb-a245-3afda0c4fc7f admin admin] RBAC: Authorizing `identity:create_grant(project_id=1b05404af98c48dd9c26aafd234846a9, user_id=1f965e6aca9341edbe6ecd391666993d, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:28.198149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a05453-af3a-48cb-a245-3afda0c4fc7f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:28.214106 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1225/2469] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:32:28 2022] PUT /identity/v3/projects/1b05404af98c48dd9c26aafd234846a9/users/1f965e6aca9341edbe6ecd391666993d/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:32:28.218069 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84325702-9c83-4e5a-b5be-8a4643f8cd5f admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:28.229425 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84325702-9c83-4e5a-b5be-8a4643f8cd5f admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:28.291691 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07aa978-494d-4ee4-9990-a20f6bb63001 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:28.292339 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07aa978-494d-4ee4-9990-a20f6bb63001 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:28.292624 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07aa978-494d-4ee4-9990-a20f6bb63001 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:28.292903 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07aa978-494d-4ee4-9990-a20f6bb63001 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:28.293587 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07aa978-494d-4ee4-9990-a20f6bb63001 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:28.295195 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07aa978-494d-4ee4-9990-a20f6bb63001 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:28.305771 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1245/2470] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:32:28 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:32:28.309940 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a05453-af3a-48cb-a245-3afda0c4fc7f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:28.320206 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a05453-af3a-48cb-a245-3afda0c4fc7f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:28.324743 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be704e7-bb8e-4a5b-a1e7-6d4369937b65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:28.325419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be704e7-bb8e-4a5b-a1e7-6d4369937b65 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:28.325697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be704e7-bb8e-4a5b-a1e7-6d4369937b65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:28.325968 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be704e7-bb8e-4a5b-a1e7-6d4369937b65 admin admin] PATH_INFO: `/v3/projects/1b05404af98c48dd9c26aafd234846a9/users/1f965e6aca9341edbe6ecd391666993d/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:28.327857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be704e7-bb8e-4a5b-a1e7-6d4369937b65 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1b05404af98c48dd9c26aafd234846a9, user_id=1f965e6aca9341edbe6ecd391666993d, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:28.330562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be704e7-bb8e-4a5b-a1e7-6d4369937b65 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:28.341486 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1226/2471] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:32:28 2022] PUT /identity/v3/projects/1b05404af98c48dd9c26aafd234846a9/users/1f965e6aca9341edbe6ecd391666993d/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:32:33.165065 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66990d3e-9f71-4ca2-b11f-7be9a59352f9 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:33.166519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66990d3e-9f71-4ca2-b11f-7be9a59352f9 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:33.166703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66990d3e-9f71-4ca2-b11f-7be9a59352f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:33.215985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-66990d3e-9f71-4ca2-b11f-7be9a59352f9 None None] MFA Rules not processed for user `1f965e6aca9341edbe6ecd391666993d`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:32:33.217293 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-66990d3e-9f71-4ca2-b11f-7be9a59352f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:33.289498 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1246/2472] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:32:33 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:32:33.303401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be704e7-bb8e-4a5b-a1e7-6d4369937b65 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:33.306177 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0be704e7-bb8e-4a5b-a1e7-6d4369937b65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:33.382636 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:32:33.410573 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be704e7-bb8e-4a5b-a1e7-6d4369937b65 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:33.417158 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b29b53b-61c3-4b53-88af-cec392b75b79 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:33.417158 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b29b53b-61c3-4b53-88af-cec392b75b79 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:33.417547 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b29b53b-61c3-4b53-88af-cec392b75b79 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:33.417833 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b29b53b-61c3-4b53-88af-cec392b75b79 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:33.489703 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b29b53b-61c3-4b53-88af-cec392b75b79 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:33.491911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b29b53b-61c3-4b53-88af-cec392b75b79 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:33.502465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1227/2473] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:32:33 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:32:34.505400 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fa00a7-014b-4417-8bf0-62002fd2293a None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:34.505909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fa00a7-014b-4417-8bf0-62002fd2293a None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:34.506027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fa00a7-014b-4417-8bf0-62002fd2293a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:34.525229 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-12fa00a7-014b-4417-8bf0-62002fd2293a None None] MFA Rules not processed for user `a71c0e86c4b2456abce0e459705a2919`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:32:34.526404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-12fa00a7-014b-4417-8bf0-62002fd2293a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:34.533184 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1247/2474] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:32:34 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:32:41.225908 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3b5d2d-cde7-4955-8dce-f7dc24f0ec81 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:41.226487 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3b5d2d-cde7-4955-8dce-f7dc24f0ec81 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:41.226487 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3b5d2d-cde7-4955-8dce-f7dc24f0ec81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:41.246230 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3f3b5d2d-cde7-4955-8dce-f7dc24f0ec81 None None] MFA Rules not processed for user `b05dcf62adc949c79b37c92bb3e09e8e`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:32:41.247589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3f3b5d2d-cde7-4955-8dce-f7dc24f0ec81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:41.308771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1228/2475] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:32:41 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:32:41.615274 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fa00a7-014b-4417-8bf0-62002fd2293a None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:41.670838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fa00a7-014b-4417-8bf0-62002fd2293a None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:41.674545 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3257f3-a808-44c5-bd4d-61cee8113d1c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:41.675160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3257f3-a808-44c5-bd4d-61cee8113d1c service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:41.675462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3257f3-a808-44c5-bd4d-61cee8113d1c service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:41.675734 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3257f3-a808-44c5-bd4d-61cee8113d1c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:41.677528 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-fe3257f3-a808-44c5-bd4d-61cee8113d1c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:41.687280 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3257f3-a808-44c5-bd4d-61cee8113d1c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:41.688707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3257f3-a808-44c5-bd4d-61cee8113d1c service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:41.697226 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1248/2476] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:32:41 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:32:43.026615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3b5d2d-cde7-4955-8dce-f7dc24f0ec81 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:43.028062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3f3b5d2d-cde7-4955-8dce-f7dc24f0ec81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:43.102228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3b5d2d-cde7-4955-8dce-f7dc24f0ec81 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:43.111937 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7bdbd7-6ae3-472d-b9d8-be9714f9ee9e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '359d79cbfcad49beba2b495066824128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:43.112787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7bdbd7-6ae3-472d-b9d8-be9714f9ee9e service placement] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:43.113148 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7bdbd7-6ae3-472d-b9d8-be9714f9ee9e service placement] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:43.113451 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7bdbd7-6ae3-472d-b9d8-be9714f9ee9e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:43.115438 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bd7bdbd7-6ae3-472d-b9d8-be9714f9ee9e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:43.128963 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7bdbd7-6ae3-472d-b9d8-be9714f9ee9e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:43.130586 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7bdbd7-6ae3-472d-b9d8-be9714f9ee9e service placement] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:43.140722 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1229/2477] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:32:43 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:32:51.693659 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=98 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:32:53.136396 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=88 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:32:56.457213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3257f3-a808-44c5-bd4d-61cee8113d1c service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:32:56.458730 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-fe3257f3-a808-44c5-bd4d-61cee8113d1c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:56.486487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3257f3-a808-44c5-bd4d-61cee8113d1c service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:32:56.505847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e2066c-454b-4276-bab9-e73c3bd8bf82 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:32:56.507156 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e2066c-454b-4276-bab9-e73c3bd8bf82 service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:56.507552 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e2066c-454b-4276-bab9-e73c3bd8bf82 service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:56.507933 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e2066c-454b-4276-bab9-e73c3bd8bf82 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:56.514714 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-54e2066c-454b-4276-bab9-e73c3bd8bf82 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:56.547379 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e2066c-454b-4276-bab9-e73c3bd8bf82 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:32:56.549194 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e2066c-454b-4276-bab9-e73c3bd8bf82 service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:32:56.563889 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1249/2478] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:32:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:32:58.481556 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8628b175-ae60-4c7b-81ad-f2a53eb7478b None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:32:58.482050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8628b175-ae60-4c7b-81ad-f2a53eb7478b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:32:58.482146 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8628b175-ae60-4c7b-81ad-f2a53eb7478b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:32:58.501030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-8628b175-ae60-4c7b-81ad-f2a53eb7478b None None] MFA Rules not processed for user `8ef7a83a87d44fc0a5647cb60057c036`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:32:58.502253 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8628b175-ae60-4c7b-81ad-f2a53eb7478b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:32:58.556412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1230/2479] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:32:58 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:06.557947 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=6 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:08.549778 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=25 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:14.976095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e2066c-454b-4276-bab9-e73c3bd8bf82 service nova] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:14.985395 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e2066c-454b-4276-bab9-e73c3bd8bf82 service nova] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:14.989222 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0ca5b4-660b-4915-adca-0195815b77c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:14.990416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0ca5b4-660b-4915-adca-0195815b77c1 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:14.990703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0ca5b4-660b-4915-adca-0195815b77c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:14.990977 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0ca5b4-660b-4915-adca-0195815b77c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:14.999743 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0ca5b4-660b-4915-adca-0195815b77c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:15.001192 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0ca5b4-660b-4915-adca-0195815b77c1 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:15.011246 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1250/2480] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:33:14 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:33:25.007032 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=6 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:26.832017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8628b175-ae60-4c7b-81ad-f2a53eb7478b None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:26.833434 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8628b175-ae60-4c7b-81ad-f2a53eb7478b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:26.845327 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8628b175-ae60-4c7b-81ad-f2a53eb7478b None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:26.848840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63620f35-7b24-47a4-9c53-eca250115974 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:26.849478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63620f35-7b24-47a4-9c53-eca250115974 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:26.849757 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63620f35-7b24-47a4-9c53-eca250115974 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:26.850034 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63620f35-7b24-47a4-9c53-eca250115974 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:26.851687 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-63620f35-7b24-47a4-9c53-eca250115974 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:26.913405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63620f35-7b24-47a4-9c53-eca250115974 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:26.914830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63620f35-7b24-47a4-9c53-eca250115974 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:26.924214 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1231/2481] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:33:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:33:29.957652 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0ca5b4-660b-4915-adca-0195815b77c1 service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:29.959151 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4e0ca5b4-660b-4915-adca-0195815b77c1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:29.985540 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0ca5b4-660b-4915-adca-0195815b77c1 service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:29.991501 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7503282-e0b7-4d7c-ac2d-105bd4636660 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:29.992355 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7503282-e0b7-4d7c-ac2d-105bd4636660 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:29.992966 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7503282-e0b7-4d7c-ac2d-105bd4636660 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:29.993483 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7503282-e0b7-4d7c-ac2d-105bd4636660 admin admin] PATH_INFO: `/v3/users/cbf574ed57d446bc8fabe7813e64e92c` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:29.994887 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7503282-e0b7-4d7c-ac2d-105bd4636660 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cbf574ed57d446bc8fabe7813e64e92c)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:29.996829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7503282-e0b7-4d7c-ac2d-105bd4636660 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:30.054390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a7503282-e0b7-4d7c-ac2d-105bd4636660 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cbf574ed57d446bc8fabe7813e64e92c'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:30.061988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a7503282-e0b7-4d7c-ac2d-105bd4636660 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cbf574ed57d446bc8fabe7813e64e92c'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:30.062657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a7503282-e0b7-4d7c-ac2d-105bd4636660 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cbf574ed57d446bc8fabe7813e64e92c'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:30.074857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a7503282-e0b7-4d7c-ac2d-105bd4636660 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cbf574ed57d446bc8fabe7813e64e92c'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:30.075463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a7503282-e0b7-4d7c-ac2d-105bd4636660 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cbf574ed57d446bc8fabe7813e64e92c'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:30.092193 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1251/2482] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:33:29 2022] DELETE /identity/v3/users/cbf574ed57d446bc8fabe7813e64e92c => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:30.266298 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63620f35-7b24-47a4-9c53-eca250115974 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:30.276844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63620f35-7b24-47a4-9c53-eca250115974 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:30.339815 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac2b708-bf49-4754-8b07-5842dc56b433 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:30.340489 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac2b708-bf49-4754-8b07-5842dc56b433 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:30.340767 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac2b708-bf49-4754-8b07-5842dc56b433 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:30.341056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac2b708-bf49-4754-8b07-5842dc56b433 admin admin] PATH_INFO: `/v3/projects/edc09fe630034be4b292562e2de6e3f9` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:30.342484 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac2b708-bf49-4754-8b07-5842dc56b433 admin admin] RBAC: Authorizing `identity:delete_project(project_id=edc09fe630034be4b292562e2de6e3f9)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:30.344500 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac2b708-bf49-4754-8b07-5842dc56b433 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:30.397758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8ac2b708-bf49-4754-8b07-5842dc56b433 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'edc09fe630034be4b292562e2de6e3f9', 'request_id': 'req-8ac2b708-bf49-4754-8b07-5842dc56b433'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:30.403886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8ac2b708-bf49-4754-8b07-5842dc56b433 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'edc09fe630034be4b292562e2de6e3f9', 'request_id': 'req-8ac2b708-bf49-4754-8b07-5842dc56b433'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:30.413512 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8ac2b708-bf49-4754-8b07-5842dc56b433 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'edc09fe630034be4b292562e2de6e3f9', 'request_id': 'req-8ac2b708-bf49-4754-8b07-5842dc56b433'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:30.425688 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8ac2b708-bf49-4754-8b07-5842dc56b433 admin admin] The token cache is being invalidate because project edc09fe630034be4b292562e2de6e3f9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:33:30.425985 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8ac2b708-bf49-4754-8b07-5842dc56b433 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:30.426664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8ac2b708-bf49-4754-8b07-5842dc56b433 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:30.428762 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1232/2483] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:33:30 2022] DELETE /identity/v3/projects/edc09fe630034be4b292562e2de6e3f9 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:30.433699 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9bd04f-7e6a-421e-be1d-d70ef6a043f5 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:30.433990 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9bd04f-7e6a-421e-be1d-d70ef6a043f5 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:30.434268 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9bd04f-7e6a-421e-be1d-d70ef6a043f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:30.454811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-1c9bd04f-7e6a-421e-be1d-d70ef6a043f5 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:30.456092 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-1c9bd04f-7e6a-421e-be1d-d70ef6a043f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:30.516324 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1252/2484] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:30.525308 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fa2e19-bf2c-4206-8e14-d46191529ffe None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:30.525599 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fa2e19-bf2c-4206-8e14-d46191529ffe None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:30.525877 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fa2e19-bf2c-4206-8e14-d46191529ffe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:30.546301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-44fa2e19-bf2c-4206-8e14-d46191529ffe None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:30.547514 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-44fa2e19-bf2c-4206-8e14-d46191529ffe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:30.555317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1233/2485] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:30.559389 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9bd04f-7e6a-421e-be1d-d70ef6a043f5 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:30.569248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9bd04f-7e6a-421e-be1d-d70ef6a043f5 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:30.573825 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d346f5-5edf-4e4c-a4b9-f20f1ac5ddce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:30.574418 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d346f5-5edf-4e4c-a4b9-f20f1ac5ddce admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:30.574697 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d346f5-5edf-4e4c-a4b9-f20f1ac5ddce admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:30.574969 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d346f5-5edf-4e4c-a4b9-f20f1ac5ddce admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:30.575610 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d346f5-5edf-4e4c-a4b9-f20f1ac5ddce admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:33:30.575917 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d346f5-5edf-4e4c-a4b9-f20f1ac5ddce admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:30.577550 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d346f5-5edf-4e4c-a4b9-f20f1ac5ddce admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:30.589737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1253/2486] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:33:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:33:30.594202 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fa2e19-bf2c-4206-8e14-d46191529ffe None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:30.604262 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fa2e19-bf2c-4206-8e14-d46191529ffe None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:30.608915 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808c7f83-d882-41a0-8f1b-13affb4600b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:30.609546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808c7f83-d882-41a0-8f1b-13affb4600b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:30.609827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808c7f83-d882-41a0-8f1b-13affb4600b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:30.610100 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808c7f83-d882-41a0-8f1b-13affb4600b1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:30.610773 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808c7f83-d882-41a0-8f1b-13affb4600b1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:30.612941 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808c7f83-d882-41a0-8f1b-13affb4600b1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:30.628271 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:30.640988 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1234/2487] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:33:30 2022] POST /identity/v3/projects => generated 371 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:33:30.645372 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d346f5-5edf-4e4c-a4b9-f20f1ac5ddce admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:30.656011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d346f5-5edf-4e4c-a4b9-f20f1ac5ddce admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:30.711613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a155ce-a5c2-43c0-b437-ea7e38075a20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:30.712219 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a155ce-a5c2-43c0-b437-ea7e38075a20 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:30.712502 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a155ce-a5c2-43c0-b437-ea7e38075a20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:30.712783 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a155ce-a5c2-43c0-b437-ea7e38075a20 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:30.713444 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a155ce-a5c2-43c0-b437-ea7e38075a20 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:30.715370 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a155ce-a5c2-43c0-b437-ea7e38075a20 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:30.723408 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:30.739814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1254/2488] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:33:30 2022] POST /identity/v3/users => generated 286 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:33:30.744087 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808c7f83-d882-41a0-8f1b-13affb4600b1 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:30.754429 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808c7f83-d882-41a0-8f1b-13affb4600b1 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:30.758962 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be29f28-759e-4712-ba3d-489af9f68fd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:30.759557 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be29f28-759e-4712-ba3d-489af9f68fd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:30.759837 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be29f28-759e-4712-ba3d-489af9f68fd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:30.760106 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be29f28-759e-4712-ba3d-489af9f68fd6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:30.760774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be29f28-759e-4712-ba3d-489af9f68fd6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:30.762504 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be29f28-759e-4712-ba3d-489af9f68fd6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:30.772842 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1235/2489] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:33:30 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:33:30.776523 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a155ce-a5c2-43c0-b437-ea7e38075a20 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:30.786752 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a155ce-a5c2-43c0-b437-ea7e38075a20 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:30.792020 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769b7da0-df16-4d74-9288-a70c4cab080a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:30.792610 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769b7da0-df16-4d74-9288-a70c4cab080a admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:30.792888 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769b7da0-df16-4d74-9288-a70c4cab080a admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:30.793190 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769b7da0-df16-4d74-9288-a70c4cab080a admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:30.793828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769b7da0-df16-4d74-9288-a70c4cab080a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:30.795419 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769b7da0-df16-4d74-9288-a70c4cab080a admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:30.807002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1255/2490] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:33:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:33:30.811157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be29f28-759e-4712-ba3d-489af9f68fd6 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:30.821584 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be29f28-759e-4712-ba3d-489af9f68fd6 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:30.826171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb405ea-a5da-41d6-a170-abc22c07f26a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:30.826814 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb405ea-a5da-41d6-a170-abc22c07f26a admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:30.827093 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb405ea-a5da-41d6-a170-abc22c07f26a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:30.827365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb405ea-a5da-41d6-a170-abc22c07f26a admin admin] PATH_INFO: `/v3/projects/55efdd317a3242218354fbb454add89d/users/6427fe3a717d4990a91e797d56c1d7a1/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:30.843177 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb405ea-a5da-41d6-a170-abc22c07f26a admin admin] RBAC: Authorizing `identity:create_grant(project_id=55efdd317a3242218354fbb454add89d, user_id=6427fe3a717d4990a91e797d56c1d7a1, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:30.845879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb405ea-a5da-41d6-a170-abc22c07f26a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:30.858689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1236/2491] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:33:30 2022] PUT /identity/v3/projects/55efdd317a3242218354fbb454add89d/users/6427fe3a717d4990a91e797d56c1d7a1/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:30.869278 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d8415d-4474-4cc2-bcd1-8b32315f2485 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:30.869563 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d8415d-4474-4cc2-bcd1-8b32315f2485 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:30.869862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d8415d-4474-4cc2-bcd1-8b32315f2485 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:30.898491 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-68d8415d-4474-4cc2-bcd1-8b32315f2485 None None] MFA Rules not processed for user `6427fe3a717d4990a91e797d56c1d7a1`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:30.899709 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-68d8415d-4474-4cc2-bcd1-8b32315f2485 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:30.962382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1256/2492] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:30 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:30.966907 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb405ea-a5da-41d6-a170-abc22c07f26a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:31.027552 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb405ea-a5da-41d6-a170-abc22c07f26a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:31.032108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3043b20-551d-4151-86fd-f61a9a65ba2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:31.032709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3043b20-551d-4151-86fd-f61a9a65ba2a admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:31.032995 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3043b20-551d-4151-86fd-f61a9a65ba2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:31.033300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3043b20-551d-4151-86fd-f61a9a65ba2a admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:31.033925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3043b20-551d-4151-86fd-f61a9a65ba2a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:31.035825 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3043b20-551d-4151-86fd-f61a9a65ba2a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:31.062177 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1237/2493] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:33:30 2022] POST /identity/v3/projects => generated 373 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:33:31.066503 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d8415d-4474-4cc2-bcd1-8b32315f2485 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:31.077558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d8415d-4474-4cc2-bcd1-8b32315f2485 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:31.134439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9afea1d-9dbc-49dc-a015-eb64e6e03663 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:31.135059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9afea1d-9dbc-49dc-a015-eb64e6e03663 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:31.135338 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9afea1d-9dbc-49dc-a015-eb64e6e03663 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:31.135611 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9afea1d-9dbc-49dc-a015-eb64e6e03663 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:31.136246 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9afea1d-9dbc-49dc-a015-eb64e6e03663 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:31.138223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9afea1d-9dbc-49dc-a015-eb64e6e03663 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:31.159997 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1257/2494] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:33:31 2022] POST /identity/v3/users => generated 293 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:33:31.164006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3043b20-551d-4151-86fd-f61a9a65ba2a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:31.174461 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3043b20-551d-4151-86fd-f61a9a65ba2a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:31.179752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda5f379-7130-4d7b-ac5e-5a4926e2747a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:31.180419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda5f379-7130-4d7b-ac5e-5a4926e2747a admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:31.180703 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda5f379-7130-4d7b-ac5e-5a4926e2747a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:31.180974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda5f379-7130-4d7b-ac5e-5a4926e2747a admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:31.181654 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda5f379-7130-4d7b-ac5e-5a4926e2747a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:31.183256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda5f379-7130-4d7b-ac5e-5a4926e2747a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:31.194272 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1238/2495] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:33:31 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:33:31.198123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9afea1d-9dbc-49dc-a015-eb64e6e03663 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:31.208230 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9afea1d-9dbc-49dc-a015-eb64e6e03663 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:31.212970 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67d903e-f755-4926-b0c5-4665238f3a49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:31.213656 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67d903e-f755-4926-b0c5-4665238f3a49 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:31.213939 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67d903e-f755-4926-b0c5-4665238f3a49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:31.214222 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67d903e-f755-4926-b0c5-4665238f3a49 admin admin] PATH_INFO: `/v3/projects/902b316638b544f883d0a9748c15c341/users/ce8540a8fa3648d0b275c24cea3f0dd2/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:31.228696 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67d903e-f755-4926-b0c5-4665238f3a49 admin admin] RBAC: Authorizing `identity:create_grant(project_id=902b316638b544f883d0a9748c15c341, user_id=ce8540a8fa3648d0b275c24cea3f0dd2, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:31.231464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67d903e-f755-4926-b0c5-4665238f3a49 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:31.245159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1258/2496] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:33:31 2022] PUT /identity/v3/projects/902b316638b544f883d0a9748c15c341/users/ce8540a8fa3648d0b275c24cea3f0dd2/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:31.249075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda5f379-7130-4d7b-ac5e-5a4926e2747a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:31.259405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda5f379-7130-4d7b-ac5e-5a4926e2747a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:31.314356 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775d4b82-4f09-43ad-b696-ae0fd204d86b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:31.314964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775d4b82-4f09-43ad-b696-ae0fd204d86b admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:31.315159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775d4b82-4f09-43ad-b696-ae0fd204d86b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:31.315432 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775d4b82-4f09-43ad-b696-ae0fd204d86b admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:31.316071 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775d4b82-4f09-43ad-b696-ae0fd204d86b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:31.317727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775d4b82-4f09-43ad-b696-ae0fd204d86b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:31.328010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1239/2497] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:33:31 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:33:31.332118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67d903e-f755-4926-b0c5-4665238f3a49 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:31.342316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67d903e-f755-4926-b0c5-4665238f3a49 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:31.346925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd55169-00c9-4cb8-9522-82f5689bf021 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:31.347569 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd55169-00c9-4cb8-9522-82f5689bf021 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:31.347851 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd55169-00c9-4cb8-9522-82f5689bf021 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:31.348175 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd55169-00c9-4cb8-9522-82f5689bf021 admin admin] PATH_INFO: `/v3/projects/902b316638b544f883d0a9748c15c341/users/ce8540a8fa3648d0b275c24cea3f0dd2/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:31.350433 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd55169-00c9-4cb8-9522-82f5689bf021 admin admin] RBAC: Authorizing `identity:create_grant(project_id=902b316638b544f883d0a9748c15c341, user_id=ce8540a8fa3648d0b275c24cea3f0dd2, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:31.353172 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd55169-00c9-4cb8-9522-82f5689bf021 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:31.363689 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1259/2498] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:33:31 2022] PUT /identity/v3/projects/902b316638b544f883d0a9748c15c341/users/ce8540a8fa3648d0b275c24cea3f0dd2/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:31.374342 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2896ff-545b-4eca-a526-411f1a1e2320 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:31.374624 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2896ff-545b-4eca-a526-411f1a1e2320 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:31.374901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2896ff-545b-4eca-a526-411f1a1e2320 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:31.402478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-fb2896ff-545b-4eca-a526-411f1a1e2320 None None] MFA Rules not processed for user `ce8540a8fa3648d0b275c24cea3f0dd2`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:31.403681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fb2896ff-545b-4eca-a526-411f1a1e2320 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:31.421313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:31.470791 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1240/2499] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:31 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:31.482202 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd55169-00c9-4cb8-9522-82f5689bf021 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:31.484017 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-0fd55169-00c9-4cb8-9522-82f5689bf021 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:31.507413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:INSERT=8,keystone:SELECT=92 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:31.543532 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd55169-00c9-4cb8-9522-82f5689bf021 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:31.547126 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1324ee0-42d1-42ee-9b40-add5d3b2db3e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:31.547732 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1324ee0-42d1-42ee-9b40-add5d3b2db3e service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:31.548118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1324ee0-42d1-42ee-9b40-add5d3b2db3e service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:31.548399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1324ee0-42d1-42ee-9b40-add5d3b2db3e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:31.557847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1324ee0-42d1-42ee-9b40-add5d3b2db3e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:31.559268 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1324ee0-42d1-42ee-9b40-add5d3b2db3e service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:31.568595 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1260/2500] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:33:31 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:33:31.944567 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2896ff-545b-4eca-a526-411f1a1e2320 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:32.006187 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2896ff-545b-4eca-a526-411f1a1e2320 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:32.010733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f40612e-fda3-413c-a66e-56493e358777 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:32.011361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f40612e-fda3-413c-a66e-56493e358777 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:32.011643 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f40612e-fda3-413c-a66e-56493e358777 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:32.011920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f40612e-fda3-413c-a66e-56493e358777 admin admin] PATH_INFO: `/v3/users/6427fe3a717d4990a91e797d56c1d7a1` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:32.013174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f40612e-fda3-413c-a66e-56493e358777 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6427fe3a717d4990a91e797d56c1d7a1)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:32.014989 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f40612e-fda3-413c-a66e-56493e358777 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:32.059503 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1f40612e-fda3-413c-a66e-56493e358777 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6427fe3a717d4990a91e797d56c1d7a1'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:32.071104 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1f40612e-fda3-413c-a66e-56493e358777 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6427fe3a717d4990a91e797d56c1d7a1'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:32.077721 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1f40612e-fda3-413c-a66e-56493e358777 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6427fe3a717d4990a91e797d56c1d7a1'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:32.085419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1f40612e-fda3-413c-a66e-56493e358777 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6427fe3a717d4990a91e797d56c1d7a1'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:32.086046 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1f40612e-fda3-413c-a66e-56493e358777 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6427fe3a717d4990a91e797d56c1d7a1'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:32.091844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1241/2501] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:33:31 2022] DELETE /identity/v3/users/6427fe3a717d4990a91e797d56c1d7a1 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:32.096273 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eaffde-38e3-40dc-9a80-b769fa7317a1 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:32.096760 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eaffde-38e3-40dc-9a80-b769fa7317a1 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:32.096864 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eaffde-38e3-40dc-9a80-b769fa7317a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:32.116664 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-19eaffde-38e3-40dc-9a80-b769fa7317a1 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:32.117912 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-19eaffde-38e3-40dc-9a80-b769fa7317a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:32.182428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1261/2502] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:32.192178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f40612e-fda3-413c-a66e-56493e358777 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:32.193613 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1f40612e-fda3-413c-a66e-56493e358777 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:32.234230 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:32.253686 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f40612e-fda3-413c-a66e-56493e358777 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:32.257901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f7bd18-7a57-459e-9716-36683e545f1a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:32.258501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f7bd18-7a57-459e-9716-36683e545f1a service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:32.258781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f7bd18-7a57-459e-9716-36683e545f1a service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:32.259055 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f7bd18-7a57-459e-9716-36683e545f1a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:32.267820 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f7bd18-7a57-459e-9716-36683e545f1a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:32.269292 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f7bd18-7a57-459e-9716-36683e545f1a service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:32.279199 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1242/2503] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:33:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:33:32.545754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19eaffde-38e3-40dc-9a80-b769fa7317a1 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:32.547230 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-19eaffde-38e3-40dc-9a80-b769fa7317a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:32.561408 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19eaffde-38e3-40dc-9a80-b769fa7317a1 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:32.566692 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0fa12d-755b-47c9-a3ba-8aa2c8cd0a23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:32.567318 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0fa12d-755b-47c9-a3ba-8aa2c8cd0a23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:32.567595 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0fa12d-755b-47c9-a3ba-8aa2c8cd0a23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:32.567866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0fa12d-755b-47c9-a3ba-8aa2c8cd0a23 admin admin] PATH_INFO: `/v3/projects/55efdd317a3242218354fbb454add89d` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:32.569728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0fa12d-755b-47c9-a3ba-8aa2c8cd0a23 admin admin] RBAC: Authorizing `identity:delete_project(project_id=55efdd317a3242218354fbb454add89d)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:32.571519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0fa12d-755b-47c9-a3ba-8aa2c8cd0a23 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:32.618740 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-0c0fa12d-755b-47c9-a3ba-8aa2c8cd0a23 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '55efdd317a3242218354fbb454add89d', 'request_id': 'req-0c0fa12d-755b-47c9-a3ba-8aa2c8cd0a23'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:32.624478 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-0c0fa12d-755b-47c9-a3ba-8aa2c8cd0a23 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '55efdd317a3242218354fbb454add89d', 'request_id': 'req-0c0fa12d-755b-47c9-a3ba-8aa2c8cd0a23'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:32.633255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-0c0fa12d-755b-47c9-a3ba-8aa2c8cd0a23 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '55efdd317a3242218354fbb454add89d', 'request_id': 'req-0c0fa12d-755b-47c9-a3ba-8aa2c8cd0a23'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:32.644245 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-0c0fa12d-755b-47c9-a3ba-8aa2c8cd0a23 admin admin] The token cache is being invalidate because project 55efdd317a3242218354fbb454add89d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:33:32.644536 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-0c0fa12d-755b-47c9-a3ba-8aa2c8cd0a23 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:32.645194 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-0c0fa12d-755b-47c9-a3ba-8aa2c8cd0a23 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:32.646676 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1262/2504] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:33:32 2022] DELETE /identity/v3/projects/55efdd317a3242218354fbb454add89d => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:32.651317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f7bd18-7a57-459e-9716-36683e545f1a service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:32.660734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f7bd18-7a57-459e-9716-36683e545f1a service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:32.716723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3808c5-1150-4a64-8eda-1269ceaf53b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:32.717421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3808c5-1150-4a64-8eda-1269ceaf53b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:32.717702 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3808c5-1150-4a64-8eda-1269ceaf53b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:32.717977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3808c5-1150-4a64-8eda-1269ceaf53b8 admin admin] PATH_INFO: `/v3/users/ce8540a8fa3648d0b275c24cea3f0dd2` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:32.719129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3808c5-1150-4a64-8eda-1269ceaf53b8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ce8540a8fa3648d0b275c24cea3f0dd2)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:32.721078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3808c5-1150-4a64-8eda-1269ceaf53b8 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:32.768078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-df3808c5-1150-4a64-8eda-1269ceaf53b8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ce8540a8fa3648d0b275c24cea3f0dd2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:32.779971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-df3808c5-1150-4a64-8eda-1269ceaf53b8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ce8540a8fa3648d0b275c24cea3f0dd2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:32.787719 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-df3808c5-1150-4a64-8eda-1269ceaf53b8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ce8540a8fa3648d0b275c24cea3f0dd2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:32.795570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-df3808c5-1150-4a64-8eda-1269ceaf53b8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ce8540a8fa3648d0b275c24cea3f0dd2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:32.796163 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-df3808c5-1150-4a64-8eda-1269ceaf53b8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ce8540a8fa3648d0b275c24cea3f0dd2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:32.801951 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1243/2505] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:33:32 2022] DELETE /identity/v3/users/ce8540a8fa3648d0b275c24cea3f0dd2 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:33.066211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0fa12d-755b-47c9-a3ba-8aa2c8cd0a23 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:33.067641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-0c0fa12d-755b-47c9-a3ba-8aa2c8cd0a23 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:33.081242 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0fa12d-755b-47c9-a3ba-8aa2c8cd0a23 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:33.127199 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:33.141549 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4c62c1-c12e-4dc7-a854-8d7f8717038b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:33.142175 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4c62c1-c12e-4dc7-a854-8d7f8717038b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:33.142451 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4c62c1-c12e-4dc7-a854-8d7f8717038b admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:33.142723 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4c62c1-c12e-4dc7-a854-8d7f8717038b admin admin] PATH_INFO: `/v3/projects/902b316638b544f883d0a9748c15c341` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:33.143825 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4c62c1-c12e-4dc7-a854-8d7f8717038b admin admin] RBAC: Authorizing `identity:delete_project(project_id=902b316638b544f883d0a9748c15c341)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:33.145834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4c62c1-c12e-4dc7-a854-8d7f8717038b admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:33.192098 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bf4c62c1-c12e-4dc7-a854-8d7f8717038b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '902b316638b544f883d0a9748c15c341', 'request_id': 'req-bf4c62c1-c12e-4dc7-a854-8d7f8717038b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:33.197909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bf4c62c1-c12e-4dc7-a854-8d7f8717038b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '902b316638b544f883d0a9748c15c341', 'request_id': 'req-bf4c62c1-c12e-4dc7-a854-8d7f8717038b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:33.204874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bf4c62c1-c12e-4dc7-a854-8d7f8717038b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '902b316638b544f883d0a9748c15c341', 'request_id': 'req-bf4c62c1-c12e-4dc7-a854-8d7f8717038b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:33.216445 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bf4c62c1-c12e-4dc7-a854-8d7f8717038b admin admin] The token cache is being invalidate because project 902b316638b544f883d0a9748c15c341 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:33:33.216742 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bf4c62c1-c12e-4dc7-a854-8d7f8717038b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:33.221594 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-bf4c62c1-c12e-4dc7-a854-8d7f8717038b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:33.223384 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1263/2506] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:33:33 2022] DELETE /identity/v3/projects/902b316638b544f883d0a9748c15c341 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:33.228781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2aef8b0-351b-4b7a-8bb3-1869a0dc4d91 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:33.229102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2aef8b0-351b-4b7a-8bb3-1869a0dc4d91 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:33.229382 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2aef8b0-351b-4b7a-8bb3-1869a0dc4d91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:33.249560 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-e2aef8b0-351b-4b7a-8bb3-1869a0dc4d91 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:33.250784 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e2aef8b0-351b-4b7a-8bb3-1869a0dc4d91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:33.300492 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:33.311913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1244/2507] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:33.319539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6dfe8d-3edf-43e9-ba35-d0b58e83bd27 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:33.319824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6dfe8d-3edf-43e9-ba35-d0b58e83bd27 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:33.320091 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6dfe8d-3edf-43e9-ba35-d0b58e83bd27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:33.340476 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-ad6dfe8d-3edf-43e9-ba35-d0b58e83bd27 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:33.341814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ad6dfe8d-3edf-43e9-ba35-d0b58e83bd27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:33.349549 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1264/2508] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:33.354199 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2aef8b0-351b-4b7a-8bb3-1869a0dc4d91 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:33.364143 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2aef8b0-351b-4b7a-8bb3-1869a0dc4d91 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:33.369439 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb62572-2e5b-44eb-ae90-508ea4611e49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:33.370084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb62572-2e5b-44eb-ae90-508ea4611e49 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:33.370391 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb62572-2e5b-44eb-ae90-508ea4611e49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:33.370725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb62572-2e5b-44eb-ae90-508ea4611e49 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:33.371368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb62572-2e5b-44eb-ae90-508ea4611e49 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:33:33.371674 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb62572-2e5b-44eb-ae90-508ea4611e49 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:33.373312 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb62572-2e5b-44eb-ae90-508ea4611e49 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:33.384744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1245/2509] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:33:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:33:33.388930 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6dfe8d-3edf-43e9-ba35-d0b58e83bd27 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:33.398546 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6dfe8d-3edf-43e9-ba35-d0b58e83bd27 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:33.403078 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a31ce00-f4ec-4ce4-89e8-027141f3ec17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:33.403656 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a31ce00-f4ec-4ce4-89e8-027141f3ec17 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:33.403930 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a31ce00-f4ec-4ce4-89e8-027141f3ec17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:33.404196 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a31ce00-f4ec-4ce4-89e8-027141f3ec17 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:33.404819 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a31ce00-f4ec-4ce4-89e8-027141f3ec17 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:33.406736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a31ce00-f4ec-4ce4-89e8-027141f3ec17 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:33.433350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1265/2510] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:33:33 2022] POST /identity/v3/projects => generated 401 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:33:33.437405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb62572-2e5b-44eb-ae90-508ea4611e49 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:33.447348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb62572-2e5b-44eb-ae90-508ea4611e49 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:33.503846 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ceb776-a06f-40a4-9a99-377c185bbd7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:33.504488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ceb776-a06f-40a4-9a99-377c185bbd7e admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:33.504770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ceb776-a06f-40a4-9a99-377c185bbd7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:33.505064 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ceb776-a06f-40a4-9a99-377c185bbd7e admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:33.505709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ceb776-a06f-40a4-9a99-377c185bbd7e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:33.507675 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ceb776-a06f-40a4-9a99-377c185bbd7e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:33.529055 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1246/2511] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:33:33 2022] POST /identity/v3/users => generated 301 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:33:33.533180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a31ce00-f4ec-4ce4-89e8-027141f3ec17 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:33.543785 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a31ce00-f4ec-4ce4-89e8-027141f3ec17 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:33.548286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6ca76e-cab2-4239-94fa-4f4562f27161 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:33.548879 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6ca76e-cab2-4239-94fa-4f4562f27161 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:33.549193 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6ca76e-cab2-4239-94fa-4f4562f27161 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:33.549468 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6ca76e-cab2-4239-94fa-4f4562f27161 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:33.550113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6ca76e-cab2-4239-94fa-4f4562f27161 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:33.551692 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6ca76e-cab2-4239-94fa-4f4562f27161 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:33.561796 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1266/2512] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:33:33 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:33:33.565400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ceb776-a06f-40a4-9a99-377c185bbd7e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:33.575130 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ceb776-a06f-40a4-9a99-377c185bbd7e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:33.579731 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8daba8-f08c-44e8-b6a2-abdc00377d1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:33.580313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8daba8-f08c-44e8-b6a2-abdc00377d1a admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:33.580598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8daba8-f08c-44e8-b6a2-abdc00377d1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:33.580874 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8daba8-f08c-44e8-b6a2-abdc00377d1a admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:33.581550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8daba8-f08c-44e8-b6a2-abdc00377d1a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:33.583138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8daba8-f08c-44e8-b6a2-abdc00377d1a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:33.594242 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1247/2513] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:33:33 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:33:33.598127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6ca76e-cab2-4239-94fa-4f4562f27161 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:33.608404 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6ca76e-cab2-4239-94fa-4f4562f27161 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:33.613244 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abc4ef5-6689-473c-87fb-ae95a282004b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:33.613873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abc4ef5-6689-473c-87fb-ae95a282004b admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:33.614304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abc4ef5-6689-473c-87fb-ae95a282004b admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:33.614588 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abc4ef5-6689-473c-87fb-ae95a282004b admin admin] PATH_INFO: `/v3/projects/f01c84583db34c64bf82909fcb8c3e99/users/d4ace8001fe54f3ab8365931487d913f/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:33.628731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abc4ef5-6689-473c-87fb-ae95a282004b admin admin] RBAC: Authorizing `identity:create_grant(project_id=f01c84583db34c64bf82909fcb8c3e99, user_id=d4ace8001fe54f3ab8365931487d913f, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:33.631354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abc4ef5-6689-473c-87fb-ae95a282004b admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:33.643554 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1267/2514] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:33:33 2022] PUT /identity/v3/projects/f01c84583db34c64bf82909fcb8c3e99/users/d4ace8001fe54f3ab8365931487d913f/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:33:33.654296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97350af-665f-447f-a828-1c96d6fa03eb None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:33.654854 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97350af-665f-447f-a828-1c96d6fa03eb None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:33.654854 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97350af-665f-447f-a828-1c96d6fa03eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:33.682899 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-f97350af-665f-447f-a828-1c96d6fa03eb None None] MFA Rules not processed for user `d4ace8001fe54f3ab8365931487d913f`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:33.684211 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f97350af-665f-447f-a828-1c96d6fa03eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:33.746496 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1248/2515] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:33 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:33.750947 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abc4ef5-6689-473c-87fb-ae95a282004b admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:33.812712 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abc4ef5-6689-473c-87fb-ae95a282004b admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:33.817411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1d01f0-7f52-44ee-8409-05472f449337 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:33.818008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1d01f0-7f52-44ee-8409-05472f449337 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:33.818286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1d01f0-7f52-44ee-8409-05472f449337 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:33.818558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1d01f0-7f52-44ee-8409-05472f449337 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:33.819717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1d01f0-7f52-44ee-8409-05472f449337 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:33.821650 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1d01f0-7f52-44ee-8409-05472f449337 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:33.841624 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:33.849294 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1268/2516] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:33:33 2022] POST /identity/v3/projects => generated 401 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:33:33.853166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97350af-665f-447f-a828-1c96d6fa03eb None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:33.863827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97350af-665f-447f-a828-1c96d6fa03eb None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:33.911349 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:33.921453 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f2a484-f2cf-458f-bf47-d4a481a261df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:33.922053 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f2a484-f2cf-458f-bf47-d4a481a261df admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:33.922333 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f2a484-f2cf-458f-bf47-d4a481a261df admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:33.922608 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f2a484-f2cf-458f-bf47-d4a481a261df admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:33.923226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f2a484-f2cf-458f-bf47-d4a481a261df admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:33.925176 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f2a484-f2cf-458f-bf47-d4a481a261df admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:33.946266 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1249/2517] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:33:33 2022] POST /identity/v3/users => generated 307 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:33:33.950129 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1d01f0-7f52-44ee-8409-05472f449337 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:33.960725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1d01f0-7f52-44ee-8409-05472f449337 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:33.965265 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbe9fc7-c558-4605-a893-cc8389529fc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:33.965847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbe9fc7-c558-4605-a893-cc8389529fc6 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:33.966120 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbe9fc7-c558-4605-a893-cc8389529fc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:33.966386 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbe9fc7-c558-4605-a893-cc8389529fc6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:33.967018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbe9fc7-c558-4605-a893-cc8389529fc6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:33.968586 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbe9fc7-c558-4605-a893-cc8389529fc6 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:33.978805 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1269/2518] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:33:33 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:33:33.982644 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f2a484-f2cf-458f-bf47-d4a481a261df admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:33.993252 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f2a484-f2cf-458f-bf47-d4a481a261df admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:33.997872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dae3e4-b88b-49ca-b4ca-70913720123b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:33.998520 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dae3e4-b88b-49ca-b4ca-70913720123b admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:33.998801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dae3e4-b88b-49ca-b4ca-70913720123b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:33.999076 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dae3e4-b88b-49ca-b4ca-70913720123b admin admin] PATH_INFO: `/v3/projects/e462700af90640a6ad42fd7126d3cf36/users/24477e645bf54298abdc31bcd603d75b/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:34.014133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14dae3e4-b88b-49ca-b4ca-70913720123b admin admin] RBAC: Authorizing `identity:create_grant(project_id=e462700af90640a6ad42fd7126d3cf36, user_id=24477e645bf54298abdc31bcd603d75b, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:34.016791 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14dae3e4-b88b-49ca-b4ca-70913720123b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:34.030073 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1250/2519] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:33:33 2022] PUT /identity/v3/projects/e462700af90640a6ad42fd7126d3cf36/users/24477e645bf54298abdc31bcd603d75b/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:34.034066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbe9fc7-c558-4605-a893-cc8389529fc6 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:34.043746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbe9fc7-c558-4605-a893-cc8389529fc6 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:34.098537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64a3cbd-5a0e-4a34-bb36-ec53ae4749bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:34.099125 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64a3cbd-5a0e-4a34-bb36-ec53ae4749bc admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:34.099401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64a3cbd-5a0e-4a34-bb36-ec53ae4749bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:34.099671 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64a3cbd-5a0e-4a34-bb36-ec53ae4749bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:34.100328 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64a3cbd-5a0e-4a34-bb36-ec53ae4749bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:34.101944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64a3cbd-5a0e-4a34-bb36-ec53ae4749bc admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:34.112040 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1270/2520] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:33:34 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:33:34.115907 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dae3e4-b88b-49ca-b4ca-70913720123b admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:34.126399 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dae3e4-b88b-49ca-b4ca-70913720123b admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:34.131058 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a654f7-7ac0-4aa0-9016-22b6cd55d69f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:34.132207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a654f7-7ac0-4aa0-9016-22b6cd55d69f admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:34.132490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a654f7-7ac0-4aa0-9016-22b6cd55d69f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:34.132766 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a654f7-7ac0-4aa0-9016-22b6cd55d69f admin admin] PATH_INFO: `/v3/projects/e462700af90640a6ad42fd7126d3cf36/users/24477e645bf54298abdc31bcd603d75b/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:34.134706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a654f7-7ac0-4aa0-9016-22b6cd55d69f admin admin] RBAC: Authorizing `identity:create_grant(project_id=e462700af90640a6ad42fd7126d3cf36, user_id=24477e645bf54298abdc31bcd603d75b, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:34.137412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a654f7-7ac0-4aa0-9016-22b6cd55d69f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:34.154291 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1251/2521] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:33:34 2022] PUT /identity/v3/projects/e462700af90640a6ad42fd7126d3cf36/users/24477e645bf54298abdc31bcd603d75b/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:34.165499 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c34c514-ae3b-46c9-802a-9b3d645289d7 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:34.165984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c34c514-ae3b-46c9-802a-9b3d645289d7 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:34.165984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c34c514-ae3b-46c9-802a-9b3d645289d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:34.194972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-5c34c514-ae3b-46c9-802a-9b3d645289d7 None None] MFA Rules not processed for user `24477e645bf54298abdc31bcd603d75b`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:34.196158 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-5c34c514-ae3b-46c9-802a-9b3d645289d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:34.261105 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1271/2522] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:34 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:34.271739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a654f7-7ac0-4aa0-9016-22b6cd55d69f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:34.273481 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-24a654f7-7ac0-4aa0-9016-22b6cd55d69f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:34.330575 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a654f7-7ac0-4aa0-9016-22b6cd55d69f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:34.334850 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7e3275-4291-47f8-91a0-bcdd1507a74b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:34.335680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7e3275-4291-47f8-91a0-bcdd1507a74b service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:34.335998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7e3275-4291-47f8-91a0-bcdd1507a74b service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:34.336381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7e3275-4291-47f8-91a0-bcdd1507a74b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:34.345946 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7e3275-4291-47f8-91a0-bcdd1507a74b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:34.347379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7e3275-4291-47f8-91a0-bcdd1507a74b service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:34.356657 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1252/2523] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:33:34 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:33:34.431943 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c34c514-ae3b-46c9-802a-9b3d645289d7 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:34.442002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c34c514-ae3b-46c9-802a-9b3d645289d7 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:34.447275 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e46a994-c5b5-42a1-b2ca-096205a91cca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:34.447961 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e46a994-c5b5-42a1-b2ca-096205a91cca service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:34.448416 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e46a994-c5b5-42a1-b2ca-096205a91cca service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:34.448617 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e46a994-c5b5-42a1-b2ca-096205a91cca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:34.504849 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e46a994-c5b5-42a1-b2ca-096205a91cca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:34.506314 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e46a994-c5b5-42a1-b2ca-096205a91cca service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:34.515049 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1272/2524] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:33:34 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:33:38.527484 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7e3275-4291-47f8-91a0-bcdd1507a74b service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:38.588376 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7e3275-4291-47f8-91a0-bcdd1507a74b service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:38.592867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a433eaf1-812f-4d3c-a9e8-205a62483573 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:38.593521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a433eaf1-812f-4d3c-a9e8-205a62483573 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:38.593802 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a433eaf1-812f-4d3c-a9e8-205a62483573 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:38.594077 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a433eaf1-812f-4d3c-a9e8-205a62483573 admin admin] PATH_INFO: `/v3/users/d4ace8001fe54f3ab8365931487d913f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:38.595191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a433eaf1-812f-4d3c-a9e8-205a62483573 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d4ace8001fe54f3ab8365931487d913f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:38.597141 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a433eaf1-812f-4d3c-a9e8-205a62483573 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:38.642489 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a433eaf1-812f-4d3c-a9e8-205a62483573 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd4ace8001fe54f3ab8365931487d913f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:38.644241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:38.656727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a433eaf1-812f-4d3c-a9e8-205a62483573 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd4ace8001fe54f3ab8365931487d913f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:38.663388 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a433eaf1-812f-4d3c-a9e8-205a62483573 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd4ace8001fe54f3ab8365931487d913f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:38.670964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a433eaf1-812f-4d3c-a9e8-205a62483573 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd4ace8001fe54f3ab8365931487d913f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:38.671614 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a433eaf1-812f-4d3c-a9e8-205a62483573 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd4ace8001fe54f3ab8365931487d913f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:38.677486 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1253/2525] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:33:38 2022] DELETE /identity/v3/users/d4ace8001fe54f3ab8365931487d913f => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:38.682000 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254506a8-dc82-42ee-9256-5c140596fdcf None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:38.682512 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254506a8-dc82-42ee-9256-5c140596fdcf None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:38.682722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254506a8-dc82-42ee-9256-5c140596fdcf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:38.696047 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:38.702427 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-254506a8-dc82-42ee-9256-5c140596fdcf None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:38.703646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-254506a8-dc82-42ee-9256-5c140596fdcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:38.761492 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1273/2526] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:38.771150 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a433eaf1-812f-4d3c-a9e8-205a62483573 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:38.772535 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a433eaf1-812f-4d3c-a9e8-205a62483573 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:38.830163 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a433eaf1-812f-4d3c-a9e8-205a62483573 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:38.833897 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c57a4b-be62-4a5e-a87b-7ffb73afcfda service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:38.834499 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c57a4b-be62-4a5e-a87b-7ffb73afcfda service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:38.834783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c57a4b-be62-4a5e-a87b-7ffb73afcfda service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:38.835059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c57a4b-be62-4a5e-a87b-7ffb73afcfda service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:38.844396 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c57a4b-be62-4a5e-a87b-7ffb73afcfda service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:38.846020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c57a4b-be62-4a5e-a87b-7ffb73afcfda service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:38.856267 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1254/2527] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:33:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:33:39.124541 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254506a8-dc82-42ee-9256-5c140596fdcf None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:39.125984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-254506a8-dc82-42ee-9256-5c140596fdcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:39.139550 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254506a8-dc82-42ee-9256-5c140596fdcf None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:39.144179 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca55670d-22f3-4e79-b341-302541aadaac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:39.144806 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca55670d-22f3-4e79-b341-302541aadaac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:39.145105 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca55670d-22f3-4e79-b341-302541aadaac admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:39.145378 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca55670d-22f3-4e79-b341-302541aadaac admin admin] PATH_INFO: `/v3/projects/f01c84583db34c64bf82909fcb8c3e99` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:39.146891 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca55670d-22f3-4e79-b341-302541aadaac admin admin] RBAC: Authorizing `identity:delete_project(project_id=f01c84583db34c64bf82909fcb8c3e99)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:39.148789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca55670d-22f3-4e79-b341-302541aadaac admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:39.195650 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ca55670d-22f3-4e79-b341-302541aadaac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f01c84583db34c64bf82909fcb8c3e99', 'request_id': 'req-ca55670d-22f3-4e79-b341-302541aadaac'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:39.204079 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ca55670d-22f3-4e79-b341-302541aadaac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f01c84583db34c64bf82909fcb8c3e99', 'request_id': 'req-ca55670d-22f3-4e79-b341-302541aadaac'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:39.210512 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ca55670d-22f3-4e79-b341-302541aadaac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f01c84583db34c64bf82909fcb8c3e99', 'request_id': 'req-ca55670d-22f3-4e79-b341-302541aadaac'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:39.222572 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ca55670d-22f3-4e79-b341-302541aadaac admin admin] The token cache is being invalidate because project f01c84583db34c64bf82909fcb8c3e99 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:33:39.222863 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ca55670d-22f3-4e79-b341-302541aadaac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:39.223496 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ca55670d-22f3-4e79-b341-302541aadaac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:39.224997 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1274/2528] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:33:39 2022] DELETE /identity/v3/projects/f01c84583db34c64bf82909fcb8c3e99 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:39.228981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c57a4b-be62-4a5e-a87b-7ffb73afcfda service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:39.238902 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c57a4b-be62-4a5e-a87b-7ffb73afcfda service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:39.295113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ccfb47-de53-4e07-b128-a0a19eb1077f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:39.295733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ccfb47-de53-4e07-b128-a0a19eb1077f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:39.296008 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ccfb47-de53-4e07-b128-a0a19eb1077f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:39.296276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ccfb47-de53-4e07-b128-a0a19eb1077f admin admin] PATH_INFO: `/v3/users/24477e645bf54298abdc31bcd603d75b` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:39.297399 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ccfb47-de53-4e07-b128-a0a19eb1077f admin admin] RBAC: Authorizing `identity:delete_user(user_id=24477e645bf54298abdc31bcd603d75b)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:39.299311 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ccfb47-de53-4e07-b128-a0a19eb1077f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:39.343475 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f8ccfb47-de53-4e07-b128-a0a19eb1077f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '24477e645bf54298abdc31bcd603d75b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:39.355238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f8ccfb47-de53-4e07-b128-a0a19eb1077f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '24477e645bf54298abdc31bcd603d75b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:39.360343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:39.363099 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f8ccfb47-de53-4e07-b128-a0a19eb1077f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '24477e645bf54298abdc31bcd603d75b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:39.372672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f8ccfb47-de53-4e07-b128-a0a19eb1077f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '24477e645bf54298abdc31bcd603d75b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:39.373332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f8ccfb47-de53-4e07-b128-a0a19eb1077f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '24477e645bf54298abdc31bcd603d75b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:39.382672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1255/2529] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:33:39 2022] DELETE /identity/v3/users/24477e645bf54298abdc31bcd603d75b => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:39.659426 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca55670d-22f3-4e79-b341-302541aadaac admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:39.660866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ca55670d-22f3-4e79-b341-302541aadaac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:39.678167 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca55670d-22f3-4e79-b341-302541aadaac admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:39.736809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c62371-bc02-4bd0-854b-fe5a835e321c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:39.737459 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c62371-bc02-4bd0-854b-fe5a835e321c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:39.737645 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c62371-bc02-4bd0-854b-fe5a835e321c admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:39.737917 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c62371-bc02-4bd0-854b-fe5a835e321c admin admin] PATH_INFO: `/v3/projects/e462700af90640a6ad42fd7126d3cf36` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:39.739023 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c62371-bc02-4bd0-854b-fe5a835e321c admin admin] RBAC: Authorizing `identity:delete_project(project_id=e462700af90640a6ad42fd7126d3cf36)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:39.741025 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c62371-bc02-4bd0-854b-fe5a835e321c admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:39.774846 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:39.788650 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-58c62371-bc02-4bd0-854b-fe5a835e321c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e462700af90640a6ad42fd7126d3cf36', 'request_id': 'req-58c62371-bc02-4bd0-854b-fe5a835e321c'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:39.794752 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-58c62371-bc02-4bd0-854b-fe5a835e321c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e462700af90640a6ad42fd7126d3cf36', 'request_id': 'req-58c62371-bc02-4bd0-854b-fe5a835e321c'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:39.802033 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-58c62371-bc02-4bd0-854b-fe5a835e321c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e462700af90640a6ad42fd7126d3cf36', 'request_id': 'req-58c62371-bc02-4bd0-854b-fe5a835e321c'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:39.814386 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-58c62371-bc02-4bd0-854b-fe5a835e321c admin admin] The token cache is being invalidate because project e462700af90640a6ad42fd7126d3cf36 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:33:39.814674 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-58c62371-bc02-4bd0-854b-fe5a835e321c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:39.815304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-58c62371-bc02-4bd0-854b-fe5a835e321c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:39.816792 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1275/2530] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:33:39 2022] DELETE /identity/v3/projects/e462700af90640a6ad42fd7126d3cf36 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:39.821689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9597dcdb-3aff-464f-85a9-7360fba63e85 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:39.822163 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9597dcdb-3aff-464f-85a9-7360fba63e85 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:39.822267 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9597dcdb-3aff-464f-85a9-7360fba63e85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:39.841313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-9597dcdb-3aff-464f-85a9-7360fba63e85 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:39.842521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9597dcdb-3aff-464f-85a9-7360fba63e85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:39.900339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1256/2531] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:39.908197 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3d6ae7-4dc9-4416-980e-b94884d2118d None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:39.908486 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3d6ae7-4dc9-4416-980e-b94884d2118d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:39.908767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3d6ae7-4dc9-4416-980e-b94884d2118d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:39.928568 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-cb3d6ae7-4dc9-4416-980e-b94884d2118d None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:39.929806 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-cb3d6ae7-4dc9-4416-980e-b94884d2118d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:39.938823 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1276/2532] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:39.943193 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9597dcdb-3aff-464f-85a9-7360fba63e85 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:39.952915 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9597dcdb-3aff-464f-85a9-7360fba63e85 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:39.957473 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac05425-4346-4fdb-9696-b1830d2898fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:39.958049 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac05425-4346-4fdb-9696-b1830d2898fc admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:39.958329 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac05425-4346-4fdb-9696-b1830d2898fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:39.958600 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac05425-4346-4fdb-9696-b1830d2898fc admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:39.959243 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac05425-4346-4fdb-9696-b1830d2898fc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:33:39.959544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac05425-4346-4fdb-9696-b1830d2898fc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:39.961160 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac05425-4346-4fdb-9696-b1830d2898fc admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:39.972479 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1257/2533] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:33:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:33:39.976541 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3d6ae7-4dc9-4416-980e-b94884d2118d None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:39.986527 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3d6ae7-4dc9-4416-980e-b94884d2118d None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:39.991248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7835cf9f-92a9-4907-9f87-af5c971662fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:39.991782 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7835cf9f-92a9-4907-9f87-af5c971662fc admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:39.992062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7835cf9f-92a9-4907-9f87-af5c971662fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:39.992333 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7835cf9f-92a9-4907-9f87-af5c971662fc admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:39.992955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7835cf9f-92a9-4907-9f87-af5c971662fc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:39.995484 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7835cf9f-92a9-4907-9f87-af5c971662fc admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:40.022496 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1277/2534] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:33:39 2022] POST /identity/v3/projects => generated 381 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:33:40.026458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac05425-4346-4fdb-9696-b1830d2898fc admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:40.036278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac05425-4346-4fdb-9696-b1830d2898fc admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:40.092037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67858ce0-5ed7-42a7-880c-9f22f3477a20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:40.092604 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67858ce0-5ed7-42a7-880c-9f22f3477a20 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:40.092882 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67858ce0-5ed7-42a7-880c-9f22f3477a20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:40.093184 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67858ce0-5ed7-42a7-880c-9f22f3477a20 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:40.093822 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67858ce0-5ed7-42a7-880c-9f22f3477a20 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:40.095734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67858ce0-5ed7-42a7-880c-9f22f3477a20 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:40.117653 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1258/2535] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:33:40 2022] POST /identity/v3/users => generated 291 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:33:40.121746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7835cf9f-92a9-4907-9f87-af5c971662fc admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:40.132215 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7835cf9f-92a9-4907-9f87-af5c971662fc admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:40.136823 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aef4fe4-018f-40fc-8fb3-4f0c1da6db05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:40.137450 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aef4fe4-018f-40fc-8fb3-4f0c1da6db05 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:40.137731 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aef4fe4-018f-40fc-8fb3-4f0c1da6db05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:40.138005 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aef4fe4-018f-40fc-8fb3-4f0c1da6db05 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:40.138647 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aef4fe4-018f-40fc-8fb3-4f0c1da6db05 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:40.140238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aef4fe4-018f-40fc-8fb3-4f0c1da6db05 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:40.150488 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1278/2536] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:33:40 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:33:40.154383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67858ce0-5ed7-42a7-880c-9f22f3477a20 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:40.164137 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67858ce0-5ed7-42a7-880c-9f22f3477a20 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:40.169303 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ecc82a5-9d01-444c-8b73-85df71796ce9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:40.169888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecc82a5-9d01-444c-8b73-85df71796ce9 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:40.170169 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecc82a5-9d01-444c-8b73-85df71796ce9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:40.170441 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecc82a5-9d01-444c-8b73-85df71796ce9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:40.171080 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ecc82a5-9d01-444c-8b73-85df71796ce9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:40.172682 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ecc82a5-9d01-444c-8b73-85df71796ce9 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:40.183136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1259/2537] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:33:40 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:33:40.187401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aef4fe4-018f-40fc-8fb3-4f0c1da6db05 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:40.197583 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aef4fe4-018f-40fc-8fb3-4f0c1da6db05 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:40.202186 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25aa8497-558d-403e-b99e-b7030d5508d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:40.202821 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25aa8497-558d-403e-b99e-b7030d5508d0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:40.203132 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25aa8497-558d-403e-b99e-b7030d5508d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:40.203406 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25aa8497-558d-403e-b99e-b7030d5508d0 admin admin] PATH_INFO: `/v3/projects/f3884b45b6204674acd9b5f83784cafc/users/14d957007493471db80e28ef353a3407/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:40.218143 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25aa8497-558d-403e-b99e-b7030d5508d0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3884b45b6204674acd9b5f83784cafc, user_id=14d957007493471db80e28ef353a3407, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:40.220777 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25aa8497-558d-403e-b99e-b7030d5508d0 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:40.233728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1279/2538] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:33:40 2022] PUT /identity/v3/projects/f3884b45b6204674acd9b5f83784cafc/users/14d957007493471db80e28ef353a3407/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:40.244140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff81ed8d-040e-493b-bffd-e6707ba83521 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:40.244374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff81ed8d-040e-493b-bffd-e6707ba83521 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:40.244647 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff81ed8d-040e-493b-bffd-e6707ba83521 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:40.271224 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ff81ed8d-040e-493b-bffd-e6707ba83521 None None] MFA Rules not processed for user `14d957007493471db80e28ef353a3407`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:40.272426 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ff81ed8d-040e-493b-bffd-e6707ba83521 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:40.315790 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:40.337437 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1260/2539] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:40 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:40.341860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25aa8497-558d-403e-b99e-b7030d5508d0 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:40.402461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25aa8497-558d-403e-b99e-b7030d5508d0 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:40.407181 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a94555c-3c49-4c25-810c-b1fe7edd7f84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:40.407775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a94555c-3c49-4c25-810c-b1fe7edd7f84 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:40.408057 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a94555c-3c49-4c25-810c-b1fe7edd7f84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:40.408334 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a94555c-3c49-4c25-810c-b1fe7edd7f84 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:40.408955 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a94555c-3c49-4c25-810c-b1fe7edd7f84 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:40.410930 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a94555c-3c49-4c25-810c-b1fe7edd7f84 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:40.437471 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1280/2540] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:33:40 2022] POST /identity/v3/projects => generated 383 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:33:40.441529 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff81ed8d-040e-493b-bffd-e6707ba83521 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:40.452722 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff81ed8d-040e-493b-bffd-e6707ba83521 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:40.509354 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1755558e-11c7-436d-95c5-ab8f9ff61ba0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:40.509921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1755558e-11c7-436d-95c5-ab8f9ff61ba0 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:40.510202 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1755558e-11c7-436d-95c5-ab8f9ff61ba0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:40.510476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1755558e-11c7-436d-95c5-ab8f9ff61ba0 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:40.511100 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1755558e-11c7-436d-95c5-ab8f9ff61ba0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:40.513013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1755558e-11c7-436d-95c5-ab8f9ff61ba0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:40.534694 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1261/2541] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:33:40 2022] POST /identity/v3/users => generated 298 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:33:40.538853 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a94555c-3c49-4c25-810c-b1fe7edd7f84 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:40.549083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a94555c-3c49-4c25-810c-b1fe7edd7f84 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:40.553941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7060d6-640b-49fb-90aa-cfba26f9d8cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:40.554566 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7060d6-640b-49fb-90aa-cfba26f9d8cd admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:40.554850 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7060d6-640b-49fb-90aa-cfba26f9d8cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:40.555124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7060d6-640b-49fb-90aa-cfba26f9d8cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:40.555766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7060d6-640b-49fb-90aa-cfba26f9d8cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:40.557411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7060d6-640b-49fb-90aa-cfba26f9d8cd admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:40.567828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1281/2542] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:33:40 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:33:40.572051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1755558e-11c7-436d-95c5-ab8f9ff61ba0 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:40.581861 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1755558e-11c7-436d-95c5-ab8f9ff61ba0 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:40.586468 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0057a3fc-539d-4e64-b9db-35384fc05f71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:40.587746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0057a3fc-539d-4e64-b9db-35384fc05f71 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:40.588023 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0057a3fc-539d-4e64-b9db-35384fc05f71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:40.588295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0057a3fc-539d-4e64-b9db-35384fc05f71 admin admin] PATH_INFO: `/v3/projects/c7c04e9d9e674085a3cc27bd2054224f/users/85fde5b6be0d4f6698cac800aa188114/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:40.602719 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0057a3fc-539d-4e64-b9db-35384fc05f71 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c7c04e9d9e674085a3cc27bd2054224f, user_id=85fde5b6be0d4f6698cac800aa188114, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:40.605467 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0057a3fc-539d-4e64-b9db-35384fc05f71 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:40.619427 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1262/2543] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:33:40 2022] PUT /identity/v3/projects/c7c04e9d9e674085a3cc27bd2054224f/users/85fde5b6be0d4f6698cac800aa188114/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:40.623276 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7060d6-640b-49fb-90aa-cfba26f9d8cd admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:40.633240 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7060d6-640b-49fb-90aa-cfba26f9d8cd admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:40.662618 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:40.690309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcb3b79-b2a5-4990-b415-0bfea6de8f64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:40.690950 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcb3b79-b2a5-4990-b415-0bfea6de8f64 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:40.691234 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcb3b79-b2a5-4990-b415-0bfea6de8f64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:40.691506 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcb3b79-b2a5-4990-b415-0bfea6de8f64 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:40.692144 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcb3b79-b2a5-4990-b415-0bfea6de8f64 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:40.693770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcb3b79-b2a5-4990-b415-0bfea6de8f64 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:40.703797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1282/2544] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:33:40 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:33:40.707869 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0057a3fc-539d-4e64-b9db-35384fc05f71 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:40.718598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0057a3fc-539d-4e64-b9db-35384fc05f71 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:40.723091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17aaad47-0ab5-4797-83b1-dd824c68648a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:40.723735 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17aaad47-0ab5-4797-83b1-dd824c68648a admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:40.724014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17aaad47-0ab5-4797-83b1-dd824c68648a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:40.724283 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17aaad47-0ab5-4797-83b1-dd824c68648a admin admin] PATH_INFO: `/v3/projects/c7c04e9d9e674085a3cc27bd2054224f/users/85fde5b6be0d4f6698cac800aa188114/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:40.726171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17aaad47-0ab5-4797-83b1-dd824c68648a admin admin] RBAC: Authorizing `identity:create_grant(project_id=c7c04e9d9e674085a3cc27bd2054224f, user_id=85fde5b6be0d4f6698cac800aa188114, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:40.728773 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17aaad47-0ab5-4797-83b1-dd824c68648a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:40.738894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1263/2545] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:33:40 2022] PUT /identity/v3/projects/c7c04e9d9e674085a3cc27bd2054224f/users/85fde5b6be0d4f6698cac800aa188114/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:40.757455 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667d8015-9e65-4fc7-b15e-e29653555199 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:40.758066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667d8015-9e65-4fc7-b15e-e29653555199 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:40.758066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667d8015-9e65-4fc7-b15e-e29653555199 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:40.786814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-667d8015-9e65-4fc7-b15e-e29653555199 None None] MFA Rules not processed for user `85fde5b6be0d4f6698cac800aa188114`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:40.788208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-667d8015-9e65-4fc7-b15e-e29653555199 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:40.855113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1283/2546] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:40 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:40.866215 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17aaad47-0ab5-4797-83b1-dd824c68648a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:40.868027 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-17aaad47-0ab5-4797-83b1-dd824c68648a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:40.925107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17aaad47-0ab5-4797-83b1-dd824c68648a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:40.928700 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df1434c-039b-47ca-9158-5ccaed4610d1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:40.929330 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df1434c-039b-47ca-9158-5ccaed4610d1 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:40.929608 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df1434c-039b-47ca-9158-5ccaed4610d1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:40.929888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df1434c-039b-47ca-9158-5ccaed4610d1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:40.939181 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df1434c-039b-47ca-9158-5ccaed4610d1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:40.940617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df1434c-039b-47ca-9158-5ccaed4610d1 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:40.950506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1264/2547] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:33:40 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:33:41.049394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667d8015-9e65-4fc7-b15e-e29653555199 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:41.059915 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667d8015-9e65-4fc7-b15e-e29653555199 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:41.063655 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8685117a-c9f1-4484-8868-cda0d6b5bba6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:41.064338 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685117a-c9f1-4484-8868-cda0d6b5bba6 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:41.064624 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685117a-c9f1-4484-8868-cda0d6b5bba6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:41.064897 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685117a-c9f1-4484-8868-cda0d6b5bba6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:41.121786 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8685117a-c9f1-4484-8868-cda0d6b5bba6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:41.123267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8685117a-c9f1-4484-8868-cda0d6b5bba6 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:41.132162 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1284/2548] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:33:41 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:33:43.688044 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df1434c-039b-47ca-9158-5ccaed4610d1 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:43.730185 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:43.751366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df1434c-039b-47ca-9158-5ccaed4610d1 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:43.755725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf1302d-f440-4328-88d2-63e8ca3ee20c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:43.756344 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf1302d-f440-4328-88d2-63e8ca3ee20c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:43.756619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf1302d-f440-4328-88d2-63e8ca3ee20c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:43.756888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf1302d-f440-4328-88d2-63e8ca3ee20c admin admin] PATH_INFO: `/v3/users/14d957007493471db80e28ef353a3407` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:43.758016 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf1302d-f440-4328-88d2-63e8ca3ee20c admin admin] RBAC: Authorizing `identity:delete_user(user_id=14d957007493471db80e28ef353a3407)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:43.759960 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf1302d-f440-4328-88d2-63e8ca3ee20c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:43.806950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-dbf1302d-f440-4328-88d2-63e8ca3ee20c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '14d957007493471db80e28ef353a3407'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:43.818711 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-dbf1302d-f440-4328-88d2-63e8ca3ee20c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '14d957007493471db80e28ef353a3407'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:43.825449 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-dbf1302d-f440-4328-88d2-63e8ca3ee20c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '14d957007493471db80e28ef353a3407'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:43.832678 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-dbf1302d-f440-4328-88d2-63e8ca3ee20c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '14d957007493471db80e28ef353a3407'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:43.833371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-dbf1302d-f440-4328-88d2-63e8ca3ee20c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '14d957007493471db80e28ef353a3407'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:43.839817 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1265/2549] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:33:43 2022] DELETE /identity/v3/users/14d957007493471db80e28ef353a3407 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:43.844403 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb090e4-8f75-4379-8553-138ad370b711 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:43.844857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb090e4-8f75-4379-8553-138ad370b711 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:43.844962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb090e4-8f75-4379-8553-138ad370b711 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:43.864116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-ffb090e4-8f75-4379-8553-138ad370b711 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:43.865340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ffb090e4-8f75-4379-8553-138ad370b711 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:43.909405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:43.924306 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1285/2550] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:33:43.934020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf1302d-f440-4328-88d2-63e8ca3ee20c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:43.935445 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-dbf1302d-f440-4328-88d2-63e8ca3ee20c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:43.992138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf1302d-f440-4328-88d2-63e8ca3ee20c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:43.995740 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a52c38-395f-4003-9a27-6f634ac50b0a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:43.996368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a52c38-395f-4003-9a27-6f634ac50b0a service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:43.996644 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a52c38-395f-4003-9a27-6f634ac50b0a service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:43.996914 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a52c38-395f-4003-9a27-6f634ac50b0a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:44.005900 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a52c38-395f-4003-9a27-6f634ac50b0a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:44.007269 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a52c38-395f-4003-9a27-6f634ac50b0a service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:44.016646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1266/2551] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:33:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:33:44.280368 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb090e4-8f75-4379-8553-138ad370b711 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:44.281840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ffb090e4-8f75-4379-8553-138ad370b711 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:44.296717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb090e4-8f75-4379-8553-138ad370b711 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:44.301453 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d32a657-a7c8-46c2-bab2-ff10733cdde2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:44.302219 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d32a657-a7c8-46c2-bab2-ff10733cdde2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:44.302368 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d32a657-a7c8-46c2-bab2-ff10733cdde2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:44.302642 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d32a657-a7c8-46c2-bab2-ff10733cdde2 admin admin] PATH_INFO: `/v3/projects/f3884b45b6204674acd9b5f83784cafc` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:44.304343 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d32a657-a7c8-46c2-bab2-ff10733cdde2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f3884b45b6204674acd9b5f83784cafc)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:44.306584 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d32a657-a7c8-46c2-bab2-ff10733cdde2 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:44.353595 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-6d32a657-a7c8-46c2-bab2-ff10733cdde2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f3884b45b6204674acd9b5f83784cafc', 'request_id': 'req-6d32a657-a7c8-46c2-bab2-ff10733cdde2'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:44.359032 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-6d32a657-a7c8-46c2-bab2-ff10733cdde2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f3884b45b6204674acd9b5f83784cafc', 'request_id': 'req-6d32a657-a7c8-46c2-bab2-ff10733cdde2'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:44.366546 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-6d32a657-a7c8-46c2-bab2-ff10733cdde2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f3884b45b6204674acd9b5f83784cafc', 'request_id': 'req-6d32a657-a7c8-46c2-bab2-ff10733cdde2'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:44.377659 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-6d32a657-a7c8-46c2-bab2-ff10733cdde2 admin admin] The token cache is being invalidate because project f3884b45b6204674acd9b5f83784cafc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:33:44.377938 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-6d32a657-a7c8-46c2-bab2-ff10733cdde2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:44.378555 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-6d32a657-a7c8-46c2-bab2-ff10733cdde2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:44.380010 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1286/2552] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:33:44 2022] DELETE /identity/v3/projects/f3884b45b6204674acd9b5f83784cafc => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:44.384472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a52c38-395f-4003-9a27-6f634ac50b0a service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:44.394037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a52c38-395f-4003-9a27-6f634ac50b0a service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:44.453000 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01180145-1dbe-4d64-a2b3-98954d1d50b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:44.453942 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01180145-1dbe-4d64-a2b3-98954d1d50b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:44.454228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01180145-1dbe-4d64-a2b3-98954d1d50b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:44.454524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01180145-1dbe-4d64-a2b3-98954d1d50b5 admin admin] PATH_INFO: `/v3/users/85fde5b6be0d4f6698cac800aa188114` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:44.455658 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01180145-1dbe-4d64-a2b3-98954d1d50b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=85fde5b6be0d4f6698cac800aa188114)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:44.457623 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01180145-1dbe-4d64-a2b3-98954d1d50b5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:44.460034 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:44.505851 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-01180145-1dbe-4d64-a2b3-98954d1d50b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '85fde5b6be0d4f6698cac800aa188114'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:44.518073 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-01180145-1dbe-4d64-a2b3-98954d1d50b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '85fde5b6be0d4f6698cac800aa188114'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:44.525481 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-01180145-1dbe-4d64-a2b3-98954d1d50b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '85fde5b6be0d4f6698cac800aa188114'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:44.532578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-01180145-1dbe-4d64-a2b3-98954d1d50b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '85fde5b6be0d4f6698cac800aa188114'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:44.533233 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-01180145-1dbe-4d64-a2b3-98954d1d50b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '85fde5b6be0d4f6698cac800aa188114'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:44.539023 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1267/2553] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:33:44 2022] DELETE /identity/v3/users/85fde5b6be0d4f6698cac800aa188114 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:44.805146 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d32a657-a7c8-46c2-bab2-ff10733cdde2 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:44.807180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-6d32a657-a7c8-46c2-bab2-ff10733cdde2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:44.820687 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d32a657-a7c8-46c2-bab2-ff10733cdde2 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:44.877031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b310b9-0cf7-4c4c-9cb8-f759fb38c2da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:44.877684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b310b9-0cf7-4c4c-9cb8-f759fb38c2da admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:44.877946 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b310b9-0cf7-4c4c-9cb8-f759fb38c2da admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:44.878211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b310b9-0cf7-4c4c-9cb8-f759fb38c2da admin admin] PATH_INFO: `/v3/projects/c7c04e9d9e674085a3cc27bd2054224f` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:44.879291 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b310b9-0cf7-4c4c-9cb8-f759fb38c2da admin admin] RBAC: Authorizing `identity:delete_project(project_id=c7c04e9d9e674085a3cc27bd2054224f)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:44.881205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b310b9-0cf7-4c4c-9cb8-f759fb38c2da admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:44.926042 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-95b310b9-0cf7-4c4c-9cb8-f759fb38c2da admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c7c04e9d9e674085a3cc27bd2054224f', 'request_id': 'req-95b310b9-0cf7-4c4c-9cb8-f759fb38c2da'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:44.932068 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-95b310b9-0cf7-4c4c-9cb8-f759fb38c2da admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c7c04e9d9e674085a3cc27bd2054224f', 'request_id': 'req-95b310b9-0cf7-4c4c-9cb8-f759fb38c2da'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:44.940067 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-95b310b9-0cf7-4c4c-9cb8-f759fb38c2da admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c7c04e9d9e674085a3cc27bd2054224f', 'request_id': 'req-95b310b9-0cf7-4c4c-9cb8-f759fb38c2da'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:44.951517 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-95b310b9-0cf7-4c4c-9cb8-f759fb38c2da admin admin] The token cache is being invalidate because project c7c04e9d9e674085a3cc27bd2054224f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:33:44.951803 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-95b310b9-0cf7-4c4c-9cb8-f759fb38c2da admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:44.952440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-95b310b9-0cf7-4c4c-9cb8-f759fb38c2da admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:44.953937 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1287/2554] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:33:44 2022] DELETE /identity/v3/projects/c7c04e9d9e674085a3cc27bd2054224f => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:44.958801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fb62f3-15ab-4792-9aad-078866dbf9dc None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:44.959270 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fb62f3-15ab-4792-9aad-078866dbf9dc None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:44.959374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fb62f3-15ab-4792-9aad-078866dbf9dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:44.979178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-00fb62f3-15ab-4792-9aad-078866dbf9dc None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:44.980361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-00fb62f3-15ab-4792-9aad-078866dbf9dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:45.038893 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1268/2555] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:45.047158 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8895167a-4132-4836-a445-301cc05b2948 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:45.047445 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8895167a-4132-4836-a445-301cc05b2948 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:45.047772 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8895167a-4132-4836-a445-301cc05b2948 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:45.056829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:45.069178 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-8895167a-4132-4836-a445-301cc05b2948 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:45.070391 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8895167a-4132-4836-a445-301cc05b2948 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:45.078286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1288/2556] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:45.082540 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fb62f3-15ab-4792-9aad-078866dbf9dc None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:45.091942 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fb62f3-15ab-4792-9aad-078866dbf9dc None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:45.096434 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9f55ef-27ec-4e05-9330-7e1aa0eae9cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:45.097020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9f55ef-27ec-4e05-9330-7e1aa0eae9cf admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:45.097325 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9f55ef-27ec-4e05-9330-7e1aa0eae9cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:45.097591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9f55ef-27ec-4e05-9330-7e1aa0eae9cf admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:45.098233 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9f55ef-27ec-4e05-9330-7e1aa0eae9cf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:33:45.098530 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9f55ef-27ec-4e05-9330-7e1aa0eae9cf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:45.100144 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9f55ef-27ec-4e05-9330-7e1aa0eae9cf admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:45.112720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1269/2557] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:33:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:33:45.116729 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8895167a-4132-4836-a445-301cc05b2948 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:45.126553 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8895167a-4132-4836-a445-301cc05b2948 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:45.131262 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b7d937-ec29-497c-8e51-69fc97be3b90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:45.131856 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b7d937-ec29-497c-8e51-69fc97be3b90 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:45.132137 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b7d937-ec29-497c-8e51-69fc97be3b90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:45.132411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b7d937-ec29-497c-8e51-69fc97be3b90 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:45.133064 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b7d937-ec29-497c-8e51-69fc97be3b90 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:45.134977 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b7d937-ec29-497c-8e51-69fc97be3b90 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:45.161734 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1289/2558] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:33:45 2022] POST /identity/v3/projects => generated 385 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:33:45.166388 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9f55ef-27ec-4e05-9330-7e1aa0eae9cf admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:45.176072 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9f55ef-27ec-4e05-9330-7e1aa0eae9cf admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:45.231530 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57c64d5-c283-4067-990a-11725a546318 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:45.232134 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57c64d5-c283-4067-990a-11725a546318 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:45.232416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57c64d5-c283-4067-990a-11725a546318 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:45.232692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57c64d5-c283-4067-990a-11725a546318 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:45.233364 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57c64d5-c283-4067-990a-11725a546318 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:45.235288 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57c64d5-c283-4067-990a-11725a546318 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:45.251608 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:45.258277 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1270/2559] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:33:45 2022] POST /identity/v3/users => generated 293 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:33:45.262776 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b7d937-ec29-497c-8e51-69fc97be3b90 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:45.272818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b7d937-ec29-497c-8e51-69fc97be3b90 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:45.277383 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a046a4b6-cad0-42cb-b4bc-dc620da7aa99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:45.277974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a046a4b6-cad0-42cb-b4bc-dc620da7aa99 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:45.278259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a046a4b6-cad0-42cb-b4bc-dc620da7aa99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:45.278529 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a046a4b6-cad0-42cb-b4bc-dc620da7aa99 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:45.279168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a046a4b6-cad0-42cb-b4bc-dc620da7aa99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:45.280751 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a046a4b6-cad0-42cb-b4bc-dc620da7aa99 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:45.291125 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1290/2560] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:33:45 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:33:45.295076 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57c64d5-c283-4067-990a-11725a546318 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:45.304935 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57c64d5-c283-4067-990a-11725a546318 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:45.310265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49bc1a68-d242-4a23-a91d-eb5c6deac3a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:45.310913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bc1a68-d242-4a23-a91d-eb5c6deac3a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:45.311191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bc1a68-d242-4a23-a91d-eb5c6deac3a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:45.311464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bc1a68-d242-4a23-a91d-eb5c6deac3a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:45.312103 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49bc1a68-d242-4a23-a91d-eb5c6deac3a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:45.313755 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49bc1a68-d242-4a23-a91d-eb5c6deac3a4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:45.324299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1271/2561] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:33:45 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:33:45.328312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a046a4b6-cad0-42cb-b4bc-dc620da7aa99 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:45.338971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a046a4b6-cad0-42cb-b4bc-dc620da7aa99 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:45.343676 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8615e2e8-70e0-497e-abe8-d78b50085d60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:45.344318 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8615e2e8-70e0-497e-abe8-d78b50085d60 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:45.344600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8615e2e8-70e0-497e-abe8-d78b50085d60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:45.344878 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8615e2e8-70e0-497e-abe8-d78b50085d60 admin admin] PATH_INFO: `/v3/projects/04c9553f22b747f5a938db2dffdb01d2/users/75e462743ec54ca2915b97c3dce6e2fa/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:45.359786 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8615e2e8-70e0-497e-abe8-d78b50085d60 admin admin] RBAC: Authorizing `identity:create_grant(project_id=04c9553f22b747f5a938db2dffdb01d2, user_id=75e462743ec54ca2915b97c3dce6e2fa, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:45.362738 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8615e2e8-70e0-497e-abe8-d78b50085d60 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:45.375447 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1291/2562] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:33:45 2022] PUT /identity/v3/projects/04c9553f22b747f5a938db2dffdb01d2/users/75e462743ec54ca2915b97c3dce6e2fa/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:45.385677 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e5475c-c37a-401d-8a9c-c38db3221b6e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:45.385964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e5475c-c37a-401d-8a9c-c38db3221b6e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:45.386240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e5475c-c37a-401d-8a9c-c38db3221b6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:45.414537 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-d3e5475c-c37a-401d-8a9c-c38db3221b6e None None] MFA Rules not processed for user `75e462743ec54ca2915b97c3dce6e2fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:45.416344 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d3e5475c-c37a-401d-8a9c-c38db3221b6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:45.479620 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1272/2563] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:45 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:33:45.484056 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8615e2e8-70e0-497e-abe8-d78b50085d60 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:45.544593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8615e2e8-70e0-497e-abe8-d78b50085d60 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:45.549120 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63a6604-4279-47cf-915a-78c2da386105 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:45.549715 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63a6604-4279-47cf-915a-78c2da386105 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:45.549988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63a6604-4279-47cf-915a-78c2da386105 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:45.550252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63a6604-4279-47cf-915a-78c2da386105 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:45.550876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63a6604-4279-47cf-915a-78c2da386105 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:45.552753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63a6604-4279-47cf-915a-78c2da386105 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:45.578511 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1292/2564] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:33:45 2022] POST /identity/v3/projects => generated 381 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:33:45.582679 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e5475c-c37a-401d-8a9c-c38db3221b6e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:45.594076 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e5475c-c37a-401d-8a9c-c38db3221b6e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:45.648832 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2f454e-eead-4e67-9be7-44892386bd23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:45.649458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2f454e-eead-4e67-9be7-44892386bd23 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:45.650253 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2f454e-eead-4e67-9be7-44892386bd23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:45.650523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2f454e-eead-4e67-9be7-44892386bd23 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:45.651152 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2f454e-eead-4e67-9be7-44892386bd23 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:45.653108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2f454e-eead-4e67-9be7-44892386bd23 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:45.673902 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1273/2565] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:33:45 2022] POST /identity/v3/users => generated 297 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:33:45.677953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63a6604-4279-47cf-915a-78c2da386105 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:45.688540 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63a6604-4279-47cf-915a-78c2da386105 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:45.693288 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111bb91c-ca24-46ed-bc24-07153bd6e281 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:45.693873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111bb91c-ca24-46ed-bc24-07153bd6e281 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:45.694164 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111bb91c-ca24-46ed-bc24-07153bd6e281 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:45.694413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111bb91c-ca24-46ed-bc24-07153bd6e281 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:45.695064 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111bb91c-ca24-46ed-bc24-07153bd6e281 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:45.697268 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111bb91c-ca24-46ed-bc24-07153bd6e281 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:45.707513 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1293/2566] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:33:45 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:33:45.711420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2f454e-eead-4e67-9be7-44892386bd23 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:45.721370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2f454e-eead-4e67-9be7-44892386bd23 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:45.725998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9534d06-46b6-4eb0-83f5-e4446ca50e36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:45.726640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9534d06-46b6-4eb0-83f5-e4446ca50e36 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:45.726921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9534d06-46b6-4eb0-83f5-e4446ca50e36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:45.727196 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9534d06-46b6-4eb0-83f5-e4446ca50e36 admin admin] PATH_INFO: `/v3/projects/7da778de311d47a499763892ca4ddc95/users/f06cd24f04324df79a68420f8cddcce5/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:45.744141 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9534d06-46b6-4eb0-83f5-e4446ca50e36 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7da778de311d47a499763892ca4ddc95, user_id=f06cd24f04324df79a68420f8cddcce5, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:45.746848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9534d06-46b6-4eb0-83f5-e4446ca50e36 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:45.759424 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1274/2567] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:33:45 2022] PUT /identity/v3/projects/7da778de311d47a499763892ca4ddc95/users/f06cd24f04324df79a68420f8cddcce5/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:45.763362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111bb91c-ca24-46ed-bc24-07153bd6e281 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:45.772973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111bb91c-ca24-46ed-bc24-07153bd6e281 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:45.829521 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75044aa9-aa82-4260-9c24-d936bc965e6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:45.830068 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75044aa9-aa82-4260-9c24-d936bc965e6e admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:45.830350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75044aa9-aa82-4260-9c24-d936bc965e6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:45.830624 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75044aa9-aa82-4260-9c24-d936bc965e6e admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:45.831257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75044aa9-aa82-4260-9c24-d936bc965e6e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:45.832848 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75044aa9-aa82-4260-9c24-d936bc965e6e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:45.842358 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1294/2568] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:33:45 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:33:45.846475 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9534d06-46b6-4eb0-83f5-e4446ca50e36 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:45.857012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9534d06-46b6-4eb0-83f5-e4446ca50e36 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:45.861913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2941856b-cb58-43bd-95f8-73923766ab73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:45.862556 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2941856b-cb58-43bd-95f8-73923766ab73 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:45.862838 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2941856b-cb58-43bd-95f8-73923766ab73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:45.863111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2941856b-cb58-43bd-95f8-73923766ab73 admin admin] PATH_INFO: `/v3/projects/7da778de311d47a499763892ca4ddc95/users/f06cd24f04324df79a68420f8cddcce5/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:45.865020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2941856b-cb58-43bd-95f8-73923766ab73 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7da778de311d47a499763892ca4ddc95, user_id=f06cd24f04324df79a68420f8cddcce5, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:45.867769 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2941856b-cb58-43bd-95f8-73923766ab73 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:45.878401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1275/2569] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:33:45 2022] PUT /identity/v3/projects/7da778de311d47a499763892ca4ddc95/users/f06cd24f04324df79a68420f8cddcce5/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:45.889450 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c050c6b-ceb8-4b3c-a389-7b110e0fc85c None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:45.889935 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c050c6b-ceb8-4b3c-a389-7b110e0fc85c None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:45.890053 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c050c6b-ceb8-4b3c-a389-7b110e0fc85c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:45.910338 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:45.919804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-8c050c6b-ceb8-4b3c-a389-7b110e0fc85c None None] MFA Rules not processed for user `f06cd24f04324df79a68420f8cddcce5`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:45.921008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8c050c6b-ceb8-4b3c-a389-7b110e0fc85c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:45.987549 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1295/2570] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:45 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:45.998183 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2941856b-cb58-43bd-95f8-73923766ab73 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:45.999998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2941856b-cb58-43bd-95f8-73923766ab73 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:46.033115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:46.060247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2941856b-cb58-43bd-95f8-73923766ab73 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:46.063927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9ef99c-8337-4003-93db-07b4d8286e80 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:46.064524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9ef99c-8337-4003-93db-07b4d8286e80 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:46.064806 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9ef99c-8337-4003-93db-07b4d8286e80 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:46.065124 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9ef99c-8337-4003-93db-07b4d8286e80 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:46.120711 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9ef99c-8337-4003-93db-07b4d8286e80 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:46.122219 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9ef99c-8337-4003-93db-07b4d8286e80 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:46.131236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1276/2571] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:33:45 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:33:50.317079 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c050c6b-ceb8-4b3c-a389-7b110e0fc85c None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:50.318500 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8c050c6b-ceb8-4b3c-a389-7b110e0fc85c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:50.330720 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c050c6b-ceb8-4b3c-a389-7b110e0fc85c None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:50.334297 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d28648-9c22-4f42-9280-76fc719a5525 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:50.334892 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d28648-9c22-4f42-9280-76fc719a5525 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:50.335169 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d28648-9c22-4f42-9280-76fc719a5525 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:50.335440 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d28648-9c22-4f42-9280-76fc719a5525 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:50.344680 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d28648-9c22-4f42-9280-76fc719a5525 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:50.346134 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d28648-9c22-4f42-9280-76fc719a5525 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:50.355789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1296/2572] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:33:50 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:33:55.209797 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9ef99c-8337-4003-93db-07b4d8286e80 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:55.271345 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9ef99c-8337-4003-93db-07b4d8286e80 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:55.275891 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e070917-e23c-4581-b058-cad063b4ade1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:55.276514 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e070917-e23c-4581-b058-cad063b4ade1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:55.276805 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e070917-e23c-4581-b058-cad063b4ade1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:55.277093 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e070917-e23c-4581-b058-cad063b4ade1 admin admin] PATH_INFO: `/v3/users/75e462743ec54ca2915b97c3dce6e2fa` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:55.278207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e070917-e23c-4581-b058-cad063b4ade1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=75e462743ec54ca2915b97c3dce6e2fa)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:55.280173 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e070917-e23c-4581-b058-cad063b4ade1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:55.329412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5e070917-e23c-4581-b058-cad063b4ade1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75e462743ec54ca2915b97c3dce6e2fa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:55.341454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5e070917-e23c-4581-b058-cad063b4ade1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75e462743ec54ca2915b97c3dce6e2fa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:55.348346 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5e070917-e23c-4581-b058-cad063b4ade1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75e462743ec54ca2915b97c3dce6e2fa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:55.355798 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5e070917-e23c-4581-b058-cad063b4ade1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75e462743ec54ca2915b97c3dce6e2fa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:55.356424 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5e070917-e23c-4581-b058-cad063b4ade1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75e462743ec54ca2915b97c3dce6e2fa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:55.362569 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1277/2573] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:33:55 2022] DELETE /identity/v3/users/75e462743ec54ca2915b97c3dce6e2fa => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:55.367056 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b529e8f6-d9e2-4427-89a2-9e649beda128 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:55.367550 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b529e8f6-d9e2-4427-89a2-9e649beda128 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:55.367661 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b529e8f6-d9e2-4427-89a2-9e649beda128 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:55.388232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-b529e8f6-d9e2-4427-89a2-9e649beda128 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:55.389467 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b529e8f6-d9e2-4427-89a2-9e649beda128 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:55.446797 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1297/2574] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:55.456676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e070917-e23c-4581-b058-cad063b4ade1 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:55.458118 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5e070917-e23c-4581-b058-cad063b4ade1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:55.463437 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:55.518242 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e070917-e23c-4581-b058-cad063b4ade1 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:55.521941 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e76aa0-c6d9-4fd1-8eb4-68d7fa5e1066 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:55.522544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e76aa0-c6d9-4fd1-8eb4-68d7fa5e1066 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:55.522823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e76aa0-c6d9-4fd1-8eb4-68d7fa5e1066 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:55.523093 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e76aa0-c6d9-4fd1-8eb4-68d7fa5e1066 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:55.532178 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e76aa0-c6d9-4fd1-8eb4-68d7fa5e1066 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:55.533651 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e76aa0-c6d9-4fd1-8eb4-68d7fa5e1066 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:55.543469 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1278/2575] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:33:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:33:55.804495 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b529e8f6-d9e2-4427-89a2-9e649beda128 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:55.806119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b529e8f6-d9e2-4427-89a2-9e649beda128 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:55.820829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b529e8f6-d9e2-4427-89a2-9e649beda128 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:55.825448 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec0b943-5151-4a81-8b0e-c2beff735f58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:55.826088 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec0b943-5151-4a81-8b0e-c2beff735f58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:55.826370 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec0b943-5151-4a81-8b0e-c2beff735f58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:55.826642 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec0b943-5151-4a81-8b0e-c2beff735f58 admin admin] PATH_INFO: `/v3/projects/04c9553f22b747f5a938db2dffdb01d2` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:55.827763 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec0b943-5151-4a81-8b0e-c2beff735f58 admin admin] RBAC: Authorizing `identity:delete_project(project_id=04c9553f22b747f5a938db2dffdb01d2)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:55.830400 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec0b943-5151-4a81-8b0e-c2beff735f58 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:55.878018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-8ec0b943-5151-4a81-8b0e-c2beff735f58 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '04c9553f22b747f5a938db2dffdb01d2', 'request_id': 'req-8ec0b943-5151-4a81-8b0e-c2beff735f58'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:55.883647 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-8ec0b943-5151-4a81-8b0e-c2beff735f58 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '04c9553f22b747f5a938db2dffdb01d2', 'request_id': 'req-8ec0b943-5151-4a81-8b0e-c2beff735f58'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:55.891155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-8ec0b943-5151-4a81-8b0e-c2beff735f58 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '04c9553f22b747f5a938db2dffdb01d2', 'request_id': 'req-8ec0b943-5151-4a81-8b0e-c2beff735f58'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:55.903622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-8ec0b943-5151-4a81-8b0e-c2beff735f58 admin admin] The token cache is being invalidate because project 04c9553f22b747f5a938db2dffdb01d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:33:55.903910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-8ec0b943-5151-4a81-8b0e-c2beff735f58 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:55.904533 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-8ec0b943-5151-4a81-8b0e-c2beff735f58 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:55.906085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1298/2576] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:33:55 2022] DELETE /identity/v3/projects/04c9553f22b747f5a938db2dffdb01d2 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:55.910016 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e76aa0-c6d9-4fd1-8eb4-68d7fa5e1066 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:55.920201 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e76aa0-c6d9-4fd1-8eb4-68d7fa5e1066 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:55.974760 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8827c53d-a858-4507-85ec-777e8a0cb1a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:55.975387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8827c53d-a858-4507-85ec-777e8a0cb1a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:55.975668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8827c53d-a858-4507-85ec-777e8a0cb1a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:55.975942 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8827c53d-a858-4507-85ec-777e8a0cb1a1 admin admin] PATH_INFO: `/v3/users/f06cd24f04324df79a68420f8cddcce5` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:55.977091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8827c53d-a858-4507-85ec-777e8a0cb1a1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f06cd24f04324df79a68420f8cddcce5)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:55.979635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8827c53d-a858-4507-85ec-777e8a0cb1a1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:56.024617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8827c53d-a858-4507-85ec-777e8a0cb1a1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f06cd24f04324df79a68420f8cddcce5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:56.036631 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8827c53d-a858-4507-85ec-777e8a0cb1a1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f06cd24f04324df79a68420f8cddcce5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:56.043875 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8827c53d-a858-4507-85ec-777e8a0cb1a1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f06cd24f04324df79a68420f8cddcce5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:56.051256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8827c53d-a858-4507-85ec-777e8a0cb1a1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f06cd24f04324df79a68420f8cddcce5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:56.051956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8827c53d-a858-4507-85ec-777e8a0cb1a1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f06cd24f04324df79a68420f8cddcce5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:56.057942 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1279/2577] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:33:55 2022] DELETE /identity/v3/users/f06cd24f04324df79a68420f8cddcce5 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:56.326428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec0b943-5151-4a81-8b0e-c2beff735f58 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:56.328047 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8ec0b943-5151-4a81-8b0e-c2beff735f58 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:56.341898 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec0b943-5151-4a81-8b0e-c2beff735f58 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:56.345732 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:56.402579 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d1bcaa-02ab-4bce-8730-98d326d4fb1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:56.403249 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d1bcaa-02ab-4bce-8730-98d326d4fb1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:56.403578 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d1bcaa-02ab-4bce-8730-98d326d4fb1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:56.403790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d1bcaa-02ab-4bce-8730-98d326d4fb1f admin admin] PATH_INFO: `/v3/projects/7da778de311d47a499763892ca4ddc95` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:56.405092 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d1bcaa-02ab-4bce-8730-98d326d4fb1f admin admin] RBAC: Authorizing `identity:delete_project(project_id=7da778de311d47a499763892ca4ddc95)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:56.407174 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d1bcaa-02ab-4bce-8730-98d326d4fb1f admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:56.452698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-41d1bcaa-02ab-4bce-8730-98d326d4fb1f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7da778de311d47a499763892ca4ddc95', 'request_id': 'req-41d1bcaa-02ab-4bce-8730-98d326d4fb1f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:56.458127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-41d1bcaa-02ab-4bce-8730-98d326d4fb1f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7da778de311d47a499763892ca4ddc95', 'request_id': 'req-41d1bcaa-02ab-4bce-8730-98d326d4fb1f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:56.465796 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-41d1bcaa-02ab-4bce-8730-98d326d4fb1f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7da778de311d47a499763892ca4ddc95', 'request_id': 'req-41d1bcaa-02ab-4bce-8730-98d326d4fb1f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:56.476567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-41d1bcaa-02ab-4bce-8730-98d326d4fb1f admin admin] The token cache is being invalidate because project 7da778de311d47a499763892ca4ddc95 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:33:56.476850 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-41d1bcaa-02ab-4bce-8730-98d326d4fb1f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:56.477500 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-41d1bcaa-02ab-4bce-8730-98d326d4fb1f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:33:56.479072 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1299/2578] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:33:56 2022] DELETE /identity/v3/projects/7da778de311d47a499763892ca4ddc95 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:56.485056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b767a4d7-39c6-4457-8429-25ea84a84fa1 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:56.485550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b767a4d7-39c6-4457-8429-25ea84a84fa1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:56.485743 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b767a4d7-39c6-4457-8429-25ea84a84fa1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:56.506471 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b767a4d7-39c6-4457-8429-25ea84a84fa1 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:56.507691 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b767a4d7-39c6-4457-8429-25ea84a84fa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:56.527175 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:56.570730 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1280/2579] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:56.580095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2312a4ed-b4fd-43bf-a6a9-2020485ab486 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:56.580519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2312a4ed-b4fd-43bf-a6a9-2020485ab486 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:56.580882 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2312a4ed-b4fd-43bf-a6a9-2020485ab486 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:56.600498 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-2312a4ed-b4fd-43bf-a6a9-2020485ab486 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:56.601722 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-2312a4ed-b4fd-43bf-a6a9-2020485ab486 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:56.610003 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1300/2580] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:56.614348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b767a4d7-39c6-4457-8429-25ea84a84fa1 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:56.624824 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b767a4d7-39c6-4457-8429-25ea84a84fa1 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:56.629533 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895a825e-87d5-4912-a68b-16025ada3a99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:56.630129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895a825e-87d5-4912-a68b-16025ada3a99 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:56.630407 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895a825e-87d5-4912-a68b-16025ada3a99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:56.630698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895a825e-87d5-4912-a68b-16025ada3a99 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:56.631347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895a825e-87d5-4912-a68b-16025ada3a99 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:33:56.631655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895a825e-87d5-4912-a68b-16025ada3a99 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:56.633283 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895a825e-87d5-4912-a68b-16025ada3a99 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:56.645442 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1281/2581] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:33:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:33:56.649830 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2312a4ed-b4fd-43bf-a6a9-2020485ab486 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:56.658984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2312a4ed-b4fd-43bf-a6a9-2020485ab486 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:56.663571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbb4d07-28b0-4983-bb64-b3e0ee5b2401 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:56.664161 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbb4d07-28b0-4983-bb64-b3e0ee5b2401 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:56.664442 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbb4d07-28b0-4983-bb64-b3e0ee5b2401 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:56.664715 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbb4d07-28b0-4983-bb64-b3e0ee5b2401 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:56.665406 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bbb4d07-28b0-4983-bb64-b3e0ee5b2401 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:56.667303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bbb4d07-28b0-4983-bb64-b3e0ee5b2401 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:56.693850 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1301/2582] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:33:56 2022] POST /identity/v3/projects => generated 377 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:33:56.698461 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895a825e-87d5-4912-a68b-16025ada3a99 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:56.708491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895a825e-87d5-4912-a68b-16025ada3a99 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:56.765389 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724b31d1-ede9-41c6-bbac-6dd9d3bde118 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:56.765982 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724b31d1-ede9-41c6-bbac-6dd9d3bde118 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:56.766259 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724b31d1-ede9-41c6-bbac-6dd9d3bde118 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:56.766527 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724b31d1-ede9-41c6-bbac-6dd9d3bde118 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:56.767169 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724b31d1-ede9-41c6-bbac-6dd9d3bde118 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:56.769104 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724b31d1-ede9-41c6-bbac-6dd9d3bde118 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:56.788455 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1282/2583] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:33:56 2022] POST /identity/v3/users => generated 289 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:33:56.793064 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbb4d07-28b0-4983-bb64-b3e0ee5b2401 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:56.803478 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbb4d07-28b0-4983-bb64-b3e0ee5b2401 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:56.808698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2056c1-d7c5-4871-acc0-2406527ec608 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:56.809332 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2056c1-d7c5-4871-acc0-2406527ec608 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:56.809613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2056c1-d7c5-4871-acc0-2406527ec608 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:56.809889 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2056c1-d7c5-4871-acc0-2406527ec608 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:56.810529 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2056c1-d7c5-4871-acc0-2406527ec608 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:56.812127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2056c1-d7c5-4871-acc0-2406527ec608 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:56.822024 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1302/2584] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:33:56 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:33:56.826321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724b31d1-ede9-41c6-bbac-6dd9d3bde118 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:56.835984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724b31d1-ede9-41c6-bbac-6dd9d3bde118 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:56.840486 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3798e5c-6baf-45a9-9b7b-b06b4a212f92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:56.841094 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3798e5c-6baf-45a9-9b7b-b06b4a212f92 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:56.841374 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3798e5c-6baf-45a9-9b7b-b06b4a212f92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:56.841641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3798e5c-6baf-45a9-9b7b-b06b4a212f92 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:56.842272 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3798e5c-6baf-45a9-9b7b-b06b4a212f92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:56.843844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3798e5c-6baf-45a9-9b7b-b06b4a212f92 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:56.855046 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1283/2585] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:33:56 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:33:56.858903 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2056c1-d7c5-4871-acc0-2406527ec608 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:56.868861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2056c1-d7c5-4871-acc0-2406527ec608 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:56.873997 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f70383-deb4-4170-ba13-af9434b450aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:56.874637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f70383-deb4-4170-ba13-af9434b450aa admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:56.874918 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f70383-deb4-4170-ba13-af9434b450aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:56.875193 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f70383-deb4-4170-ba13-af9434b450aa admin admin] PATH_INFO: `/v3/projects/c214d93ad3434fdf8929eefbf5151e53/users/ce1c37b03d9744b68e345f87510e3f4a/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:56.890024 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f70383-deb4-4170-ba13-af9434b450aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=c214d93ad3434fdf8929eefbf5151e53, user_id=ce1c37b03d9744b68e345f87510e3f4a, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:56.892692 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f70383-deb4-4170-ba13-af9434b450aa admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:56.906166 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1303/2586] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:33:56 2022] PUT /identity/v3/projects/c214d93ad3434fdf8929eefbf5151e53/users/ce1c37b03d9744b68e345f87510e3f4a/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:33:56.971330 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2801e0b6-5add-471e-8df5-c7ca03ee3e2e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:56.971626 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2801e0b6-5add-471e-8df5-c7ca03ee3e2e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:56.971899 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2801e0b6-5add-471e-8df5-c7ca03ee3e2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:57.000110 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-2801e0b6-5add-471e-8df5-c7ca03ee3e2e None None] MFA Rules not processed for user `ce1c37b03d9744b68e345f87510e3f4a`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:33:57.001345 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2801e0b6-5add-471e-8df5-c7ca03ee3e2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:57.063351 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1284/2587] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:33:56 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:33:57.073573 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f70383-deb4-4170-ba13-af9434b450aa admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:57.075289 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2801e0b6-5add-471e-8df5-c7ca03ee3e2e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:33:57.075662 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f8f70383-deb4-4170-ba13-af9434b450aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:57.077162 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2801e0b6-5add-471e-8df5-c7ca03ee3e2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:57.099202 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:57.113897 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:33:57.142011 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2801e0b6-5add-471e-8df5-c7ca03ee3e2e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:57.145712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447338b2-de34-4ade-92ca-2dc003957bf4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:57.146343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447338b2-de34-4ade-92ca-2dc003957bf4 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:57.146906 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447338b2-de34-4ade-92ca-2dc003957bf4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:57.147189 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447338b2-de34-4ade-92ca-2dc003957bf4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:57.155436 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447338b2-de34-4ade-92ca-2dc003957bf4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:57.156875 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f70383-deb4-4170-ba13-af9434b450aa admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:33:57.157014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447338b2-de34-4ade-92ca-2dc003957bf4 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:57.162728 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9cdceb-5fb0-413e-a9f0-4fbffdc9c2fa service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:33:57.163411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9cdceb-5fb0-413e-a9f0-4fbffdc9c2fa service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:33:57.163702 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9cdceb-5fb0-413e-a9f0-4fbffdc9c2fa service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:33:57.163974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9cdceb-5fb0-413e-a9f0-4fbffdc9c2fa service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:33:57.165579 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1285/2588] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:33:57 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:33:57.165717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-bb9cdceb-5fb0-413e-a9f0-4fbffdc9c2fa service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:33:57.225825 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9cdceb-5fb0-413e-a9f0-4fbffdc9c2fa service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:33:57.227342 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9cdceb-5fb0-413e-a9f0-4fbffdc9c2fa service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:33:57.237253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1304/2589] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:33:57 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:34:07.162745 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=17 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:34:07.232672 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=41 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:34:18.838107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447338b2-de34-4ade-92ca-2dc003957bf4 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:18.944004 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447338b2-de34-4ade-92ca-2dc003957bf4 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:18.948623 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b80271-68ff-4dd9-ac94-366f77a6971d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:18.949289 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b80271-68ff-4dd9-ac94-366f77a6971d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:18.949579 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b80271-68ff-4dd9-ac94-366f77a6971d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:18.949914 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b80271-68ff-4dd9-ac94-366f77a6971d admin admin] PATH_INFO: `/v3/users/ce1c37b03d9744b68e345f87510e3f4a` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:18.951544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b80271-68ff-4dd9-ac94-366f77a6971d admin admin] RBAC: Authorizing `identity:delete_user(user_id=ce1c37b03d9744b68e345f87510e3f4a)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:18.953510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b80271-68ff-4dd9-ac94-366f77a6971d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:19.001581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-71b80271-68ff-4dd9-ac94-366f77a6971d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ce1c37b03d9744b68e345f87510e3f4a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:19.013987 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-71b80271-68ff-4dd9-ac94-366f77a6971d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ce1c37b03d9744b68e345f87510e3f4a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:19.021129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-71b80271-68ff-4dd9-ac94-366f77a6971d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ce1c37b03d9744b68e345f87510e3f4a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:19.029141 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-71b80271-68ff-4dd9-ac94-366f77a6971d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ce1c37b03d9744b68e345f87510e3f4a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:19.029940 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-71b80271-68ff-4dd9-ac94-366f77a6971d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ce1c37b03d9744b68e345f87510e3f4a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:19.035974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1286/2590] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:34:18 2022] DELETE /identity/v3/users/ce1c37b03d9744b68e345f87510e3f4a => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:34:19.040510 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d704fd90-bca1-4b68-93c8-d9f52636cdf7 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:19.041067 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d704fd90-bca1-4b68-93c8-d9f52636cdf7 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:19.041204 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d704fd90-bca1-4b68-93c8-d9f52636cdf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:19.091456 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-d704fd90-bca1-4b68-93c8-d9f52636cdf7 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:19.092771 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d704fd90-bca1-4b68-93c8-d9f52636cdf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:19.158884 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1305/2591] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:34:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:34:19.169008 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b80271-68ff-4dd9-ac94-366f77a6971d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:19.170427 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-71b80271-68ff-4dd9-ac94-366f77a6971d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:19.210950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:34:19.244408 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b80271-68ff-4dd9-ac94-366f77a6971d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:19.248900 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1696dbe-307b-4a26-adf4-d8ce5215b0d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:19.249531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1696dbe-307b-4a26-adf4-d8ce5215b0d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:19.249813 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1696dbe-307b-4a26-adf4-d8ce5215b0d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:19.250085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1696dbe-307b-4a26-adf4-d8ce5215b0d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:19.258861 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1696dbe-307b-4a26-adf4-d8ce5215b0d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:19.260280 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1696dbe-307b-4a26-adf4-d8ce5215b0d4 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:19.269759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1287/2592] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:34:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:34:19.572500 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d704fd90-bca1-4b68-93c8-d9f52636cdf7 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:19.574035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d704fd90-bca1-4b68-93c8-d9f52636cdf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:19.586283 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d704fd90-bca1-4b68-93c8-d9f52636cdf7 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:19.591616 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a923ac-54a2-4e94-a32a-8e0e7e9754af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:19.592241 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a923ac-54a2-4e94-a32a-8e0e7e9754af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:19.592522 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a923ac-54a2-4e94-a32a-8e0e7e9754af admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:19.592820 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a923ac-54a2-4e94-a32a-8e0e7e9754af admin admin] PATH_INFO: `/v3/projects/c214d93ad3434fdf8929eefbf5151e53` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:19.594043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a923ac-54a2-4e94-a32a-8e0e7e9754af admin admin] RBAC: Authorizing `identity:delete_project(project_id=c214d93ad3434fdf8929eefbf5151e53)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:19.596060 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a923ac-54a2-4e94-a32a-8e0e7e9754af admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:19.644593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e7a923ac-54a2-4e94-a32a-8e0e7e9754af admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c214d93ad3434fdf8929eefbf5151e53', 'request_id': 'req-e7a923ac-54a2-4e94-a32a-8e0e7e9754af'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:19.650301 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e7a923ac-54a2-4e94-a32a-8e0e7e9754af admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c214d93ad3434fdf8929eefbf5151e53', 'request_id': 'req-e7a923ac-54a2-4e94-a32a-8e0e7e9754af'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:19.657622 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e7a923ac-54a2-4e94-a32a-8e0e7e9754af admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c214d93ad3434fdf8929eefbf5151e53', 'request_id': 'req-e7a923ac-54a2-4e94-a32a-8e0e7e9754af'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:19.668310 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e7a923ac-54a2-4e94-a32a-8e0e7e9754af admin admin] The token cache is being invalidate because project c214d93ad3434fdf8929eefbf5151e53 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:34:19.669272 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e7a923ac-54a2-4e94-a32a-8e0e7e9754af admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:19.669886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e7a923ac-54a2-4e94-a32a-8e0e7e9754af admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:19.671341 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1306/2593] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:34:19 2022] DELETE /identity/v3/projects/c214d93ad3434fdf8929eefbf5151e53 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:34:19.677385 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f773c595-6838-44c7-b0da-39b6828565c2 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:19.677674 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f773c595-6838-44c7-b0da-39b6828565c2 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:19.677951 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f773c595-6838-44c7-b0da-39b6828565c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:19.725871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-f773c595-6838-44c7-b0da-39b6828565c2 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:19.727068 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f773c595-6838-44c7-b0da-39b6828565c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:19.784873 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1288/2594] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:34:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:34:19.793476 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9a5621-2590-4026-b76b-a0134064fb58 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:19.793840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9a5621-2590-4026-b76b-a0134064fb58 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:19.794113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9a5621-2590-4026-b76b-a0134064fb58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:19.813619 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-ff9a5621-2590-4026-b76b-a0134064fb58 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:19.814821 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ff9a5621-2590-4026-b76b-a0134064fb58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:19.822707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1307/2595] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:34:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:34:19.827443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f773c595-6838-44c7-b0da-39b6828565c2 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:19.837022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f773c595-6838-44c7-b0da-39b6828565c2 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:19.842111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc35830-595b-4a2d-8726-761ddbcb8f5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:19.842689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc35830-595b-4a2d-8726-761ddbcb8f5e admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:19.842962 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc35830-595b-4a2d-8726-761ddbcb8f5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:19.843229 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc35830-595b-4a2d-8726-761ddbcb8f5e admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:19.843870 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc35830-595b-4a2d-8726-761ddbcb8f5e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:34:19.844168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc35830-595b-4a2d-8726-761ddbcb8f5e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:19.845783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc35830-595b-4a2d-8726-761ddbcb8f5e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:19.857226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1289/2596] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:34:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:34:19.861692 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9a5621-2590-4026-b76b-a0134064fb58 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:19.870788 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9a5621-2590-4026-b76b-a0134064fb58 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:19.875378 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f208ec-ce77-4ef0-9bae-7ff23ee57ee6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:19.875964 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f208ec-ce77-4ef0-9bae-7ff23ee57ee6 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:19.876240 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f208ec-ce77-4ef0-9bae-7ff23ee57ee6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:19.876511 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f208ec-ce77-4ef0-9bae-7ff23ee57ee6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:19.877165 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f208ec-ce77-4ef0-9bae-7ff23ee57ee6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:19.879048 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f208ec-ce77-4ef0-9bae-7ff23ee57ee6 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:19.906580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1308/2597] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:34:19 2022] POST /identity/v3/projects => generated 373 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:34:19.910590 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc35830-595b-4a2d-8726-761ddbcb8f5e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:19.920431 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc35830-595b-4a2d-8726-761ddbcb8f5e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:19.975123 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18e75fa-b84e-4217-ba2d-1a3fc3578224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:19.975715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18e75fa-b84e-4217-ba2d-1a3fc3578224 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:19.976021 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18e75fa-b84e-4217-ba2d-1a3fc3578224 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:19.976289 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18e75fa-b84e-4217-ba2d-1a3fc3578224 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:19.976910 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18e75fa-b84e-4217-ba2d-1a3fc3578224 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:19.978844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18e75fa-b84e-4217-ba2d-1a3fc3578224 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:20.000194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1290/2598] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:34:19 2022] POST /identity/v3/users => generated 287 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:34:20.004265 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f208ec-ce77-4ef0-9bae-7ff23ee57ee6 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:20.015159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f208ec-ce77-4ef0-9bae-7ff23ee57ee6 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:20.019804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703e4151-ccac-45e6-a0b8-1b7d2e17b4a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:20.020394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703e4151-ccac-45e6-a0b8-1b7d2e17b4a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:20.020676 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703e4151-ccac-45e6-a0b8-1b7d2e17b4a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:20.020948 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703e4151-ccac-45e6-a0b8-1b7d2e17b4a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:20.021620 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703e4151-ccac-45e6-a0b8-1b7d2e17b4a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:20.023196 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703e4151-ccac-45e6-a0b8-1b7d2e17b4a2 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:20.033715 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1309/2599] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:34:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:34:20.037640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18e75fa-b84e-4217-ba2d-1a3fc3578224 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:20.047769 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18e75fa-b84e-4217-ba2d-1a3fc3578224 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:20.052397 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b7f64e-621c-4e51-a5ff-2d8330f94f0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:20.052984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b7f64e-621c-4e51-a5ff-2d8330f94f0c admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:20.053311 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b7f64e-621c-4e51-a5ff-2d8330f94f0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:20.053570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b7f64e-621c-4e51-a5ff-2d8330f94f0c admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:20.054206 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b7f64e-621c-4e51-a5ff-2d8330f94f0c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:20.055790 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b7f64e-621c-4e51-a5ff-2d8330f94f0c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:20.067378 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1291/2600] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:34:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:34:20.071269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703e4151-ccac-45e6-a0b8-1b7d2e17b4a2 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:20.081512 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703e4151-ccac-45e6-a0b8-1b7d2e17b4a2 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:20.086119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5948c74b-5626-4eb6-a109-d923426436fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:20.086763 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5948c74b-5626-4eb6-a109-d923426436fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:20.087044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5948c74b-5626-4eb6-a109-d923426436fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:20.087324 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5948c74b-5626-4eb6-a109-d923426436fa admin admin] PATH_INFO: `/v3/projects/37b888e14f724a60bdb1fef1a4aa7cd7/users/cfdc2d9a039a4fa6bf04d2cc41b1db57/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:20.102159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5948c74b-5626-4eb6-a109-d923426436fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=37b888e14f724a60bdb1fef1a4aa7cd7, user_id=cfdc2d9a039a4fa6bf04d2cc41b1db57, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:20.104815 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5948c74b-5626-4eb6-a109-d923426436fa admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:20.107264 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:34:20.122826 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1310/2601] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:34:20 2022] PUT /identity/v3/projects/37b888e14f724a60bdb1fef1a4aa7cd7/users/cfdc2d9a039a4fa6bf04d2cc41b1db57/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:34:20.134424 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a350c71e-1c34-461f-baf3-63b6dbaf2c3f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:20.134730 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a350c71e-1c34-461f-baf3-63b6dbaf2c3f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:20.135029 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a350c71e-1c34-461f-baf3-63b6dbaf2c3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:20.148890 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:34:20.163589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-a350c71e-1c34-461f-baf3-63b6dbaf2c3f None None] MFA Rules not processed for user `cfdc2d9a039a4fa6bf04d2cc41b1db57`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:20.164809 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a350c71e-1c34-461f-baf3-63b6dbaf2c3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:20.227380 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1292/2602] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:34:20 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:34:20.238179 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5948c74b-5626-4eb6-a109-d923426436fa admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:20.239827 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-5948c74b-5626-4eb6-a109-d923426436fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:20.296298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5948c74b-5626-4eb6-a109-d923426436fa admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:20.299945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4743b30-6fbe-482b-9ee0-d29666397e3a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:20.300612 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4743b30-6fbe-482b-9ee0-d29666397e3a service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:20.300888 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4743b30-6fbe-482b-9ee0-d29666397e3a service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:20.301199 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4743b30-6fbe-482b-9ee0-d29666397e3a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:20.309985 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4743b30-6fbe-482b-9ee0-d29666397e3a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:20.311420 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4743b30-6fbe-482b-9ee0-d29666397e3a service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:20.320294 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1311/2603] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:34:20 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:34:21.686860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fb22dc-16fc-45d0-8433-e1bcb923c627 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:21.687338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fb22dc-16fc-45d0-8433-e1bcb923c627 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:21.687451 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fb22dc-16fc-45d0-8433-e1bcb923c627 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:21.706611 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-01fb22dc-16fc-45d0-8433-e1bcb923c627 None None] MFA Rules not processed for user `cfdc2d9a039a4fa6bf04d2cc41b1db57`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:21.707811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-01fb22dc-16fc-45d0-8433-e1bcb923c627 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:21.714548 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1293/2604] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:34:21 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:34:22.445315 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd0b7b1-2d66-4cbe-a009-378b40d2803d None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:22.445846 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd0b7b1-2d66-4cbe-a009-378b40d2803d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:22.445846 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd0b7b1-2d66-4cbe-a009-378b40d2803d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:22.476625 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-ffd0b7b1-2d66-4cbe-a009-378b40d2803d None None] MFA Rules not processed for user `00aeab772672403984bdcb8baaa1520e`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:22.477864 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ffd0b7b1-2d66-4cbe-a009-378b40d2803d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:22.546755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1312/2605] 10.222.0.85 () {58 vars in 1009 bytes} [Tue Aug 30 04:34:22 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:34:22.596676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01fb22dc-16fc-45d0-8433-e1bcb923c627 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:22.598195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-01fb22dc-16fc-45d0-8433-e1bcb923c627 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:22.672215 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01fb22dc-16fc-45d0-8433-e1bcb923c627 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:22.678786 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294ce8c1-7293-49cc-8011-d422067dd4b4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cb539ef6a69406baa70f660e1a815b4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:22.679399 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294ce8c1-7293-49cc-8011-d422067dd4b4 service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:22.679677 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294ce8c1-7293-49cc-8011-d422067dd4b4 service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:22.680014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294ce8c1-7293-49cc-8011-d422067dd4b4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:22.689171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294ce8c1-7293-49cc-8011-d422067dd4b4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:22.690468 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294ce8c1-7293-49cc-8011-d422067dd4b4 service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:22.699083 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1294/2606] 10.222.0.85 () {60 vars in 1384 bytes} [Tue Aug 30 04:34:22 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 04:34:22.748395 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd0b7b1-2d66-4cbe-a009-378b40d2803d None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:22.749924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ffd0b7b1-2d66-4cbe-a009-378b40d2803d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:22.883924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd0b7b1-2d66-4cbe-a009-378b40d2803d None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:22.890571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf632b62-8aa0-4165-a05f-04108fc4a29b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '359d79cbfcad49beba2b495066824128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:22.891203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf632b62-8aa0-4165-a05f-04108fc4a29b service placement] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:22.891939 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf632b62-8aa0-4165-a05f-04108fc4a29b service placement] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:22.892221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf632b62-8aa0-4165-a05f-04108fc4a29b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:22.895649 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-cf632b62-8aa0-4165-a05f-04108fc4a29b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:22.908223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:34:22.913329 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf632b62-8aa0-4165-a05f-04108fc4a29b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:22.915836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf632b62-8aa0-4165-a05f-04108fc4a29b service placement] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:22.929439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1313/2607] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:34:22 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:34:32.695354 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=64 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:34:32.926025 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:34:41.432824 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294ce8c1-7293-49cc-8011-d422067dd4b4 service swift] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:41.434437 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-294ce8c1-7293-49cc-8011-d422067dd4b4 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:41.522725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294ce8c1-7293-49cc-8011-d422067dd4b4 service swift] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:41.526358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3c2c6c-3346-456c-9e9c-84b409438d33 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:41.526968 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3c2c6c-3346-456c-9e9c-84b409438d33 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:41.527248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3c2c6c-3346-456c-9e9c-84b409438d33 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:41.527523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3c2c6c-3346-456c-9e9c-84b409438d33 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:41.530078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3c3c2c6c-3346-456c-9e9c-84b409438d33 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:41.540079 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3c2c6c-3346-456c-9e9c-84b409438d33 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:41.541594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3c2c6c-3346-456c-9e9c-84b409438d33 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:41.549894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1295/2608] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:34:41 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:34:41.688786 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0299ef22-9d25-4428-9ff8-72393fb8cbab None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:41.689413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0299ef22-9d25-4428-9ff8-72393fb8cbab None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:41.689413 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0299ef22-9d25-4428-9ff8-72393fb8cbab None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:41.691168 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1314/2609] 10.222.0.85 () {54 vars in 921 bytes} [Tue Aug 30 04:34:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:34:41.701977 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9832c9e-d474-4b6d-8d95-b5557bdcd80f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:41.702339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9832c9e-d474-4b6d-8d95-b5557bdcd80f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:41.702739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9832c9e-d474-4b6d-8d95-b5557bdcd80f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:41.740913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-f9832c9e-d474-4b6d-8d95-b5557bdcd80f None None] MFA Rules not processed for user `4bc4d455ebf041d488b1199ca58358a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:41.742392 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f9832c9e-d474-4b6d-8d95-b5557bdcd80f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:41.830815 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1296/2610] 10.222.0.85 () {58 vars in 1018 bytes} [Tue Aug 30 04:34:41 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:34:41.852205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0299ef22-9d25-4428-9ff8-72393fb8cbab None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:41.853967 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-0299ef22-9d25-4428-9ff8-72393fb8cbab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:41.920257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0299ef22-9d25-4428-9ff8-72393fb8cbab None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:41.924519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd94405f-4496-4ec8-81c9-44e935216c1b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '354b8228661c4369b3bca2e631f3f140', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:41.925232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd94405f-4496-4ec8-81c9-44e935216c1b service glance] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:41.925581 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd94405f-4496-4ec8-81c9-44e935216c1b service glance] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:41.925922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd94405f-4496-4ec8-81c9-44e935216c1b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:41.939019 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd94405f-4496-4ec8-81c9-44e935216c1b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:41.940639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd94405f-4496-4ec8-81c9-44e935216c1b service glance] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:41.953356 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1315/2611] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:34:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:34:42.015690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0753222b-6cf4-4fba-a8c6-4e0348bc3fbb None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:42.016024 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0753222b-6cf4-4fba-a8c6-4e0348bc3fbb None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:42.016360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0753222b-6cf4-4fba-a8c6-4e0348bc3fbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:42.024999 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26219769-db7e-4968-a580-15a65be16c4c None None] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:42.025325 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26219769-db7e-4968-a580-15a65be16c4c None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:42.025620 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26219769-db7e-4968-a580-15a65be16c4c None None] PATH_INFO: `/` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:42.027029 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1316/2612] 10.222.0.85 () {54 vars in 921 bytes} [Tue Aug 30 04:34:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 04:34:42.031538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e0eb95-a225-4a0a-a722-fe30193c7ef8 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:42.031880 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e0eb95-a225-4a0a-a722-fe30193c7ef8 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:42.032182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e0eb95-a225-4a0a-a722-fe30193c7ef8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:42.036640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-0753222b-6cf4-4fba-a8c6-4e0348bc3fbb None None] MFA Rules not processed for user `00aeab772672403984bdcb8baaa1520e`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:42.037890 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0753222b-6cf4-4fba-a8c6-4e0348bc3fbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:42.045097 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1297/2613] 10.222.0.85 () {58 vars in 1009 bytes} [Tue Aug 30 04:34:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:34:42.052312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-f2e0eb95-a225-4a0a-a722-fe30193c7ef8 None None] MFA Rules not processed for user `5c2edd77896a4534888d6758ec0ee0bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:42.052954 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f2e0eb95-a225-4a0a-a722-fe30193c7ef8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:42.057730 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0753222b-6cf4-4fba-a8c6-4e0348bc3fbb None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:42.060530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1317/2614] 10.222.0.85 () {58 vars in 1018 bytes} [Tue Aug 30 04:34:42 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:34:42.070794 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0753222b-6cf4-4fba-a8c6-4e0348bc3fbb None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:42.072877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e0eb95-a225-4a0a-a722-fe30193c7ef8 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:42.076459 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70cc7437-b8e6-4a22-89af-3f399fb61ef3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cb539ef6a69406baa70f660e1a815b4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:42.077082 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cc7437-b8e6-4a22-89af-3f399fb61ef3 service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:42.077372 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cc7437-b8e6-4a22-89af-3f399fb61ef3 service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:42.077846 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cc7437-b8e6-4a22-89af-3f399fb61ef3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:42.082492 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e0eb95-a225-4a0a-a722-fe30193c7ef8 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:42.086367 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44efe7d-b3b3-4b6c-b751-2d0a00548504 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:42.086519 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70cc7437-b8e6-4a22-89af-3f399fb61ef3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:42.086959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44efe7d-b3b3-4b6c-b751-2d0a00548504 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:42.087239 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44efe7d-b3b3-4b6c-b751-2d0a00548504 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:42.087539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44efe7d-b3b3-4b6c-b751-2d0a00548504 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:42.088100 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70cc7437-b8e6-4a22-89af-3f399fb61ef3 service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:42.094851 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44efe7d-b3b3-4b6c-b751-2d0a00548504 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:42.096160 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1298/2615] 10.222.0.85 () {60 vars in 1384 bytes} [Tue Aug 30 04:34:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 30 04:34:42.096284 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44efe7d-b3b3-4b6c-b751-2d0a00548504 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:42.110428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1318/2616] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:34:42 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:34:42.209291 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa11f6e-6624-41f0-bf38-4cfe233e11c3 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:42.209617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa11f6e-6624-41f0-bf38-4cfe233e11c3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:42.209876 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa11f6e-6624-41f0-bf38-4cfe233e11c3 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:42.211575 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1299/2617] 10.222.0.85 () {54 vars in 921 bytes} [Tue Aug 30 04:34:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 04:34:42.216259 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775db783-82c9-483a-a111-25ee105b5529 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:42.216534 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775db783-82c9-483a-a111-25ee105b5529 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:42.216929 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775db783-82c9-483a-a111-25ee105b5529 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:42.242656 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-775db783-82c9-483a-a111-25ee105b5529 None None] MFA Rules not processed for user `4bc4d455ebf041d488b1199ca58358a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:42.243864 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-775db783-82c9-483a-a111-25ee105b5529 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:42.253651 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1319/2618] 10.222.0.85 () {58 vars in 1018 bytes} [Tue Aug 30 04:34:42 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:34:42.264732 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa11f6e-6624-41f0-bf38-4cfe233e11c3 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:42.266949 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9fa11f6e-6624-41f0-bf38-4cfe233e11c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:42.282251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa11f6e-6624-41f0-bf38-4cfe233e11c3 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:42.286085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78204e06-3f66-44ff-b3c2-514a050557a7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:42.286761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78204e06-3f66-44ff-b3c2-514a050557a7 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:42.287109 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78204e06-3f66-44ff-b3c2-514a050557a7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:42.287453 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78204e06-3f66-44ff-b3c2-514a050557a7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:42.303781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78204e06-3f66-44ff-b3c2-514a050557a7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:42.305632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78204e06-3f66-44ff-b3c2-514a050557a7 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:42.321457 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1300/2619] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:34:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:34:42.642920 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ac0661-ce5e-41e3-8cd2-216de891872f None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:42.643331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ac0661-ce5e-41e3-8cd2-216de891872f None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:42.643703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ac0661-ce5e-41e3-8cd2-216de891872f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:42.664820 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-12ac0661-ce5e-41e3-8cd2-216de891872f None None] MFA Rules not processed for user `00aeab772672403984bdcb8baaa1520e`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:42.666142 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-12ac0661-ce5e-41e3-8cd2-216de891872f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:42.675182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1320/2620] 10.222.0.85 () {58 vars in 967 bytes} [Tue Aug 30 04:34:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:34:42.689632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78204e06-3f66-44ff-b3c2-514a050557a7 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:42.701866 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78204e06-3f66-44ff-b3c2-514a050557a7 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:42.707280 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f11eb49-8377-4f81-a485-fed7652c3c88 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cb539ef6a69406baa70f660e1a815b4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:42.707818 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f11eb49-8377-4f81-a485-fed7652c3c88 service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:42.708102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f11eb49-8377-4f81-a485-fed7652c3c88 service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:42.708379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f11eb49-8377-4f81-a485-fed7652c3c88 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:42.716837 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f11eb49-8377-4f81-a485-fed7652c3c88 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:42.718484 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f11eb49-8377-4f81-a485-fed7652c3c88 service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:42.726551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1301/2621] 10.222.0.85 () {60 vars in 1384 bytes} [Tue Aug 30 04:34:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 04:34:42.987990 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba27e169-56eb-4141-a3aa-370dc4a35390 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:42.988514 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba27e169-56eb-4141-a3aa-370dc4a35390 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:42.988740 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba27e169-56eb-4141-a3aa-370dc4a35390 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:43.014381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-ba27e169-56eb-4141-a3aa-370dc4a35390 None None] MFA Rules not processed for user `00aeab772672403984bdcb8baaa1520e`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:43.016290 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ba27e169-56eb-4141-a3aa-370dc4a35390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:43.027718 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1321/2622] 10.222.0.85 () {58 vars in 1009 bytes} [Tue Aug 30 04:34:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:34:43.043239 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f11eb49-8377-4f81-a485-fed7652c3c88 service swift] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:43.053708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f11eb49-8377-4f81-a485-fed7652c3c88 service swift] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:43.058608 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5cbf5a-1874-4622-ad4e-b05c3dada375 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cb539ef6a69406baa70f660e1a815b4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:43.059228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5cbf5a-1874-4622-ad4e-b05c3dada375 service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:43.059601 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5cbf5a-1874-4622-ad4e-b05c3dada375 service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:43.060161 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5cbf5a-1874-4622-ad4e-b05c3dada375 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:43.067050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:34:43.069983 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5cbf5a-1874-4622-ad4e-b05c3dada375 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:43.071852 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5cbf5a-1874-4622-ad4e-b05c3dada375 service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:43.080953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1302/2623] 10.222.0.85 () {60 vars in 1384 bytes} [Tue Aug 30 04:34:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 04:34:43.404918 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ef47e3-4185-4cac-8650-f3e6d0936d79 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:43.405238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ef47e3-4185-4cac-8650-f3e6d0936d79 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:43.405514 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ef47e3-4185-4cac-8650-f3e6d0936d79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:43.423983 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-f9ef47e3-4185-4cac-8650-f3e6d0936d79 None None] MFA Rules not processed for user `00aeab772672403984bdcb8baaa1520e`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:43.425193 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f9ef47e3-4185-4cac-8650-f3e6d0936d79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:43.432136 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1322/2624] 10.222.0.85 () {58 vars in 967 bytes} [Tue Aug 30 04:34:43 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:34:43.445681 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5cbf5a-1874-4622-ad4e-b05c3dada375 service swift] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:43.456439 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5cbf5a-1874-4622-ad4e-b05c3dada375 service swift] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:43.461384 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2a53c6-cd0e-40ae-8ad1-eb9bfde1c9ef service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cb539ef6a69406baa70f660e1a815b4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:43.461974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2a53c6-cd0e-40ae-8ad1-eb9bfde1c9ef service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:43.462253 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2a53c6-cd0e-40ae-8ad1-eb9bfde1c9ef service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:43.462523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2a53c6-cd0e-40ae-8ad1-eb9bfde1c9ef service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:43.470568 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2a53c6-cd0e-40ae-8ad1-eb9bfde1c9ef service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:43.472074 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2a53c6-cd0e-40ae-8ad1-eb9bfde1c9ef service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:43.479893 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1303/2625] 10.222.0.85 () {60 vars in 1384 bytes} [Tue Aug 30 04:34:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 04:34:44.399434 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51762a95-255a-43ad-91f5-222aa3ee991a None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:44.399962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51762a95-255a-43ad-91f5-222aa3ee991a None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:44.399962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51762a95-255a-43ad-91f5-222aa3ee991a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:44.423390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-51762a95-255a-43ad-91f5-222aa3ee991a None None] MFA Rules not processed for user `00aeab772672403984bdcb8baaa1520e`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:44.424726 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-51762a95-255a-43ad-91f5-222aa3ee991a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:44.432001 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1323/2626] 10.222.0.85 () {58 vars in 1009 bytes} [Tue Aug 30 04:34:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:34:44.450628 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2a53c6-cd0e-40ae-8ad1-eb9bfde1c9ef service swift] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:44.466876 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2a53c6-cd0e-40ae-8ad1-eb9bfde1c9ef service swift] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:44.474647 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24b5a97-0cef-484b-950a-9c8a5b9a3e41 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cb539ef6a69406baa70f660e1a815b4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:44.475559 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24b5a97-0cef-484b-950a-9c8a5b9a3e41 service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:44.476047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24b5a97-0cef-484b-950a-9c8a5b9a3e41 service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:44.476577 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24b5a97-0cef-484b-950a-9c8a5b9a3e41 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:44.486665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24b5a97-0cef-484b-950a-9c8a5b9a3e41 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:44.488368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24b5a97-0cef-484b-950a-9c8a5b9a3e41 service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:44.496807 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1304/2627] 10.222.0.85 () {60 vars in 1384 bytes} [Tue Aug 30 04:34:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 30 04:34:45.610360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51762a95-255a-43ad-91f5-222aa3ee991a None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:45.680690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51762a95-255a-43ad-91f5-222aa3ee991a None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:45.685317 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fcadcd-10a9-4bae-b065-eec244b6c311 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:45.686167 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fcadcd-10a9-4bae-b065-eec244b6c311 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:45.686618 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fcadcd-10a9-4bae-b065-eec244b6c311 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:45.687050 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fcadcd-10a9-4bae-b065-eec244b6c311 admin admin] PATH_INFO: `/v3/users/cfdc2d9a039a4fa6bf04d2cc41b1db57` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:45.688489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97fcadcd-10a9-4bae-b065-eec244b6c311 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cfdc2d9a039a4fa6bf04d2cc41b1db57)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:45.690629 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97fcadcd-10a9-4bae-b065-eec244b6c311 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:45.743593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-97fcadcd-10a9-4bae-b065-eec244b6c311 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cfdc2d9a039a4fa6bf04d2cc41b1db57'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:45.750756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:34:45.756236 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-97fcadcd-10a9-4bae-b065-eec244b6c311 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cfdc2d9a039a4fa6bf04d2cc41b1db57'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:45.757613 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-97fcadcd-10a9-4bae-b065-eec244b6c311 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cfdc2d9a039a4fa6bf04d2cc41b1db57'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:45.773582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-97fcadcd-10a9-4bae-b065-eec244b6c311 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cfdc2d9a039a4fa6bf04d2cc41b1db57'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:45.774351 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-97fcadcd-10a9-4bae-b065-eec244b6c311 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cfdc2d9a039a4fa6bf04d2cc41b1db57'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:45.788295 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1324/2628] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:34:45 2022] DELETE /identity/v3/users/cfdc2d9a039a4fa6bf04d2cc41b1db57 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:34:45.793449 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75ff4b6-e20d-4fc2-85f8-34cadd5dff80 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:45.794568 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75ff4b6-e20d-4fc2-85f8-34cadd5dff80 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:45.794568 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75ff4b6-e20d-4fc2-85f8-34cadd5dff80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:45.818241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c75ff4b6-e20d-4fc2-85f8-34cadd5dff80 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:45.819512 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c75ff4b6-e20d-4fc2-85f8-34cadd5dff80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:45.889719 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1305/2629] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:34:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:34:45.900888 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fcadcd-10a9-4bae-b065-eec244b6c311 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:45.902424 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-97fcadcd-10a9-4bae-b065-eec244b6c311 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:45.964691 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fcadcd-10a9-4bae-b065-eec244b6c311 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:45.969051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757d3f90-62c2-4699-8d0e-96463a316e3c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:45.969786 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757d3f90-62c2-4699-8d0e-96463a316e3c service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:45.970137 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757d3f90-62c2-4699-8d0e-96463a316e3c service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:45.970472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757d3f90-62c2-4699-8d0e-96463a316e3c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:45.983523 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757d3f90-62c2-4699-8d0e-96463a316e3c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:45.985995 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757d3f90-62c2-4699-8d0e-96463a316e3c service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:45.999384 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1325/2630] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:34:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:34:46.191301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75ff4b6-e20d-4fc2-85f8-34cadd5dff80 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:46.192801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c75ff4b6-e20d-4fc2-85f8-34cadd5dff80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:46.220338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75ff4b6-e20d-4fc2-85f8-34cadd5dff80 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:46.226166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d148415-ca59-4a7b-ae32-9eb429aa8488 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:46.226849 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d148415-ca59-4a7b-ae32-9eb429aa8488 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:46.227154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d148415-ca59-4a7b-ae32-9eb429aa8488 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:46.227382 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d148415-ca59-4a7b-ae32-9eb429aa8488 admin admin] PATH_INFO: `/v3/projects/37b888e14f724a60bdb1fef1a4aa7cd7` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:46.228506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d148415-ca59-4a7b-ae32-9eb429aa8488 admin admin] RBAC: Authorizing `identity:delete_project(project_id=37b888e14f724a60bdb1fef1a4aa7cd7)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:46.230601 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d148415-ca59-4a7b-ae32-9eb429aa8488 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:46.292544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7d148415-ca59-4a7b-ae32-9eb429aa8488 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '37b888e14f724a60bdb1fef1a4aa7cd7', 'request_id': 'req-7d148415-ca59-4a7b-ae32-9eb429aa8488'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:46.299582 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7d148415-ca59-4a7b-ae32-9eb429aa8488 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '37b888e14f724a60bdb1fef1a4aa7cd7', 'request_id': 'req-7d148415-ca59-4a7b-ae32-9eb429aa8488'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:46.307564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7d148415-ca59-4a7b-ae32-9eb429aa8488 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '37b888e14f724a60bdb1fef1a4aa7cd7', 'request_id': 'req-7d148415-ca59-4a7b-ae32-9eb429aa8488'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:46.319919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7d148415-ca59-4a7b-ae32-9eb429aa8488 admin admin] The token cache is being invalidate because project 37b888e14f724a60bdb1fef1a4aa7cd7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:34:46.320972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7d148415-ca59-4a7b-ae32-9eb429aa8488 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:46.321758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7d148415-ca59-4a7b-ae32-9eb429aa8488 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:34:46.323406 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1306/2631] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:34:46 2022] DELETE /identity/v3/projects/37b888e14f724a60bdb1fef1a4aa7cd7 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:34:46.329156 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dddbd8-c27d-4400-b257-09b30e5b7acc None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:46.329512 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dddbd8-c27d-4400-b257-09b30e5b7acc None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:46.329845 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dddbd8-c27d-4400-b257-09b30e5b7acc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:46.353053 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-b7dddbd8-c27d-4400-b257-09b30e5b7acc None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:46.354381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b7dddbd8-c27d-4400-b257-09b30e5b7acc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:46.419516 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1326/2632] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:34:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:34:46.427952 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6aaef99-df59-4f8a-a211-e23590073811 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:46.428241 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6aaef99-df59-4f8a-a211-e23590073811 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:46.428537 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6aaef99-df59-4f8a-a211-e23590073811 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:46.450312 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c6aaef99-df59-4f8a-a211-e23590073811 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:46.455137 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c6aaef99-df59-4f8a-a211-e23590073811 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:46.466998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1307/2633] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:34:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:34:46.468026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dddbd8-c27d-4400-b257-09b30e5b7acc None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:46.482302 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dddbd8-c27d-4400-b257-09b30e5b7acc None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:46.488110 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9d1a69-2cf9-43df-8432-7cdfbfacf838 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:46.488850 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9d1a69-2cf9-43df-8432-7cdfbfacf838 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:46.489344 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9d1a69-2cf9-43df-8432-7cdfbfacf838 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:46.489727 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9d1a69-2cf9-43df-8432-7cdfbfacf838 admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:46.490360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9d1a69-2cf9-43df-8432-7cdfbfacf838 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:34:46.490660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9d1a69-2cf9-43df-8432-7cdfbfacf838 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:46.492263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9d1a69-2cf9-43df-8432-7cdfbfacf838 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:46.512752 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1327/2634] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:34:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:34:46.514171 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6aaef99-df59-4f8a-a211-e23590073811 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:46.524060 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6aaef99-df59-4f8a-a211-e23590073811 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:46.529289 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d7690d-e6ab-43a6-bb3f-e4f6bed3a1bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:46.530022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d7690d-e6ab-43a6-bb3f-e4f6bed3a1bb admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:46.530366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d7690d-e6ab-43a6-bb3f-e4f6bed3a1bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:46.530707 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d7690d-e6ab-43a6-bb3f-e4f6bed3a1bb admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:46.531421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d7690d-e6ab-43a6-bb3f-e4f6bed3a1bb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:46.536076 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d7690d-e6ab-43a6-bb3f-e4f6bed3a1bb admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:46.569670 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1308/2635] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:34:46 2022] POST /identity/v3/projects => generated 377 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:34:46.570479 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9d1a69-2cf9-43df-8432-7cdfbfacf838 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:46.582902 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9d1a69-2cf9-43df-8432-7cdfbfacf838 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:46.673874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f0b6df-7020-4e7f-938d-55f4a69a4953 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:46.675219 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f0b6df-7020-4e7f-938d-55f4a69a4953 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:46.675670 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f0b6df-7020-4e7f-938d-55f4a69a4953 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:46.676094 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f0b6df-7020-4e7f-938d-55f4a69a4953 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:46.676966 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f0b6df-7020-4e7f-938d-55f4a69a4953 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:46.679120 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f0b6df-7020-4e7f-938d-55f4a69a4953 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:46.688987 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:34:46.707768 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1328/2636] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:34:46 2022] POST /identity/v3/users => generated 289 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:34:46.712073 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d7690d-e6ab-43a6-bb3f-e4f6bed3a1bb admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:46.723354 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d7690d-e6ab-43a6-bb3f-e4f6bed3a1bb admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:46.729732 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaae9093-1f67-46f1-a91b-845542b0bb05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:46.729981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaae9093-1f67-46f1-a91b-845542b0bb05 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:46.730284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaae9093-1f67-46f1-a91b-845542b0bb05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:46.730420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaae9093-1f67-46f1-a91b-845542b0bb05 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:46.732535 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaae9093-1f67-46f1-a91b-845542b0bb05 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:46.736188 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaae9093-1f67-46f1-a91b-845542b0bb05 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:46.754809 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f0b6df-7020-4e7f-938d-55f4a69a4953 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:46.755954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1309/2637] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:34:46 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:34:46.772813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f0b6df-7020-4e7f-938d-55f4a69a4953 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:46.777892 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32b1ceb-587a-46f5-b657-bc4523f13c54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:46.778508 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32b1ceb-587a-46f5-b657-bc4523f13c54 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:46.778787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32b1ceb-587a-46f5-b657-bc4523f13c54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:46.779059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32b1ceb-587a-46f5-b657-bc4523f13c54 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:46.785763 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32b1ceb-587a-46f5-b657-bc4523f13c54 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:46.787548 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32b1ceb-587a-46f5-b657-bc4523f13c54 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:46.801787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1329/2638] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:34:46 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:34:46.807426 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaae9093-1f67-46f1-a91b-845542b0bb05 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:46.839607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaae9093-1f67-46f1-a91b-845542b0bb05 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:46.844938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513fa9db-3344-454c-9bcb-ac897168a218 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:46.846066 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513fa9db-3344-454c-9bcb-ac897168a218 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:46.846588 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513fa9db-3344-454c-9bcb-ac897168a218 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:46.846944 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513fa9db-3344-454c-9bcb-ac897168a218 admin admin] PATH_INFO: `/v3/projects/18e4bf7297124da4b43ccaeda048f772/users/2625939a0b9a43d59452df9ba9982ecb/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:46.861749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:34:46.868592 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-513fa9db-3344-454c-9bcb-ac897168a218 admin admin] RBAC: Authorizing `identity:create_grant(project_id=18e4bf7297124da4b43ccaeda048f772, user_id=2625939a0b9a43d59452df9ba9982ecb, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:46.871979 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-513fa9db-3344-454c-9bcb-ac897168a218 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:46.895635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1310/2639] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:34:46 2022] PUT /identity/v3/projects/18e4bf7297124da4b43ccaeda048f772/users/2625939a0b9a43d59452df9ba9982ecb/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:34:46.900836 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76545523-25cd-4671-9537-cda7007c566e None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:46.901426 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76545523-25cd-4671-9537-cda7007c566e None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:46.901942 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76545523-25cd-4671-9537-cda7007c566e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:46.923396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-76545523-25cd-4671-9537-cda7007c566e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:46.925074 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-76545523-25cd-4671-9537-cda7007c566e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:47.000500 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1330/2640] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:34:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:34:47.007447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513fa9db-3344-454c-9bcb-ac897168a218 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:47.014741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-513fa9db-3344-454c-9bcb-ac897168a218 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:47.103986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513fa9db-3344-454c-9bcb-ac897168a218 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:47.108604 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef19ac76-7c53-4958-883c-f780bc801414 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:47.109536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef19ac76-7c53-4958-883c-f780bc801414 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:47.110148 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef19ac76-7c53-4958-883c-f780bc801414 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:47.110488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef19ac76-7c53-4958-883c-f780bc801414 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:47.126525 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef19ac76-7c53-4958-883c-f780bc801414 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:47.128272 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef19ac76-7c53-4958-883c-f780bc801414 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:47.144688 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1311/2641] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:34:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:34:52.607651 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f410c4b8-7dd7-4362-8de5-36bd19053cbb None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:52.608358 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f410c4b8-7dd7-4362-8de5-36bd19053cbb None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:52.608358 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f410c4b8-7dd7-4362-8de5-36bd19053cbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:52.641802 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-f410c4b8-7dd7-4362-8de5-36bd19053cbb None None] MFA Rules not processed for user `2625939a0b9a43d59452df9ba9982ecb`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:52.643133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f410c4b8-7dd7-4362-8de5-36bd19053cbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:52.750383 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1331/2642] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:34:52 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:34:52.754890 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef19ac76-7c53-4958-883c-f780bc801414 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:52.765030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef19ac76-7c53-4958-883c-f780bc801414 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:52.769754 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd7bdcb-b83e-4217-b037-00867d00354b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:52.770418 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd7bdcb-b83e-4217-b037-00867d00354b admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:52.770699 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd7bdcb-b83e-4217-b037-00867d00354b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:52.770969 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd7bdcb-b83e-4217-b037-00867d00354b admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:52.771612 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd7bdcb-b83e-4217-b037-00867d00354b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:52.773591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd7bdcb-b83e-4217-b037-00867d00354b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:52.810142 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1312/2643] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:34:52 2022] POST /identity/v3/projects => generated 379 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:34:52.811296 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f410c4b8-7dd7-4362-8de5-36bd19053cbb None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:52.822511 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f410c4b8-7dd7-4362-8de5-36bd19053cbb None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:52.881576 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25b9159-feac-48d9-ad73-227d11e96eaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:52.882193 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25b9159-feac-48d9-ad73-227d11e96eaa admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:52.882469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25b9159-feac-48d9-ad73-227d11e96eaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:52.882739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25b9159-feac-48d9-ad73-227d11e96eaa admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:52.883366 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25b9159-feac-48d9-ad73-227d11e96eaa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:52.885399 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25b9159-feac-48d9-ad73-227d11e96eaa admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:52.902438 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:34:52.913733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd7bdcb-b83e-4217-b037-00867d00354b admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:52.914386 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1332/2644] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:34:52 2022] POST /identity/v3/users => generated 296 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:34:52.925706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd7bdcb-b83e-4217-b037-00867d00354b admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:52.930571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87deaf33-f763-4288-a7f7-4556d1b7cd74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:52.931197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87deaf33-f763-4288-a7f7-4556d1b7cd74 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:52.931482 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87deaf33-f763-4288-a7f7-4556d1b7cd74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:52.931756 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87deaf33-f763-4288-a7f7-4556d1b7cd74 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:52.932412 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87deaf33-f763-4288-a7f7-4556d1b7cd74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:52.934632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87deaf33-f763-4288-a7f7-4556d1b7cd74 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:52.947240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1313/2645] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:34:52 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:34:52.951673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25b9159-feac-48d9-ad73-227d11e96eaa admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:52.963169 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25b9159-feac-48d9-ad73-227d11e96eaa admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:52.971858 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78865d22-3a71-4f9c-b850-61d83653162d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:52.972542 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78865d22-3a71-4f9c-b850-61d83653162d admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:52.972825 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78865d22-3a71-4f9c-b850-61d83653162d admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:52.973134 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78865d22-3a71-4f9c-b850-61d83653162d admin admin] PATH_INFO: `/v3/projects/af87dd63dbe545c3a99753b771eb2cfc/users/8ce000ee58dc4352a91e77ec52fcb26b/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:52.991095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78865d22-3a71-4f9c-b850-61d83653162d admin admin] RBAC: Authorizing `identity:create_grant(project_id=af87dd63dbe545c3a99753b771eb2cfc, user_id=8ce000ee58dc4352a91e77ec52fcb26b, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:52.993787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78865d22-3a71-4f9c-b850-61d83653162d admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:53.014476 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1333/2646] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:34:52 2022] PUT /identity/v3/projects/af87dd63dbe545c3a99753b771eb2cfc/users/8ce000ee58dc4352a91e77ec52fcb26b/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:34:53.018524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87deaf33-f763-4288-a7f7-4556d1b7cd74 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:53.029336 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87deaf33-f763-4288-a7f7-4556d1b7cd74 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:53.088422 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d2c972-7c1c-431e-b0bb-29266078cb53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:53.089635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d2c972-7c1c-431e-b0bb-29266078cb53 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:53.090514 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d2c972-7c1c-431e-b0bb-29266078cb53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:53.090879 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d2c972-7c1c-431e-b0bb-29266078cb53 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:53.091834 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d2c972-7c1c-431e-b0bb-29266078cb53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:53.093526 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d2c972-7c1c-431e-b0bb-29266078cb53 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:53.114822 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1314/2647] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:34:53 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:34:53.118662 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78865d22-3a71-4f9c-b850-61d83653162d admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:53.137535 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78865d22-3a71-4f9c-b850-61d83653162d admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:53.143915 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2077cda9-9b9b-4e7b-a9ba-4f1684dfbcda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:53.144588 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2077cda9-9b9b-4e7b-a9ba-4f1684dfbcda admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:53.144866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2077cda9-9b9b-4e7b-a9ba-4f1684dfbcda admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:53.145205 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2077cda9-9b9b-4e7b-a9ba-4f1684dfbcda admin admin] PATH_INFO: `/v3/projects/af87dd63dbe545c3a99753b771eb2cfc/users/8ce000ee58dc4352a91e77ec52fcb26b/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:53.147632 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2077cda9-9b9b-4e7b-a9ba-4f1684dfbcda admin admin] RBAC: Authorizing `identity:create_grant(project_id=af87dd63dbe545c3a99753b771eb2cfc, user_id=8ce000ee58dc4352a91e77ec52fcb26b, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:53.151930 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2077cda9-9b9b-4e7b-a9ba-4f1684dfbcda admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:53.163469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1334/2648] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:34:53 2022] PUT /identity/v3/projects/af87dd63dbe545c3a99753b771eb2cfc/users/8ce000ee58dc4352a91e77ec52fcb26b/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:34:58.238853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf56453d-291f-47b5-a95c-21f23cffe955 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:58.239402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf56453d-291f-47b5-a95c-21f23cffe955 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:58.239402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf56453d-291f-47b5-a95c-21f23cffe955 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:58.298655 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-cf56453d-291f-47b5-a95c-21f23cffe955 None None] MFA Rules not processed for user `8ce000ee58dc4352a91e77ec52fcb26b`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:58.299921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cf56453d-291f-47b5-a95c-21f23cffe955 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:58.365094 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:34:58.393934 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1315/2649] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:34:58 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:34:58.409825 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2077cda9-9b9b-4e7b-a9ba-4f1684dfbcda admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:58.413748 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-2077cda9-9b9b-4e7b-a9ba-4f1684dfbcda admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:58.516519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2077cda9-9b9b-4e7b-a9ba-4f1684dfbcda admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:58.523276 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d2b2e5-ecaa-4011-bb75-6f3babfee4d1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:58.524488 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d2b2e5-ecaa-4011-bb75-6f3babfee4d1 service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:58.524773 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d2b2e5-ecaa-4011-bb75-6f3babfee4d1 service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:58.525070 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d2b2e5-ecaa-4011-bb75-6f3babfee4d1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:58.600233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d2b2e5-ecaa-4011-bb75-6f3babfee4d1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:58.602083 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d2b2e5-ecaa-4011-bb75-6f3babfee4d1 service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:58.612636 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1335/2650] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:34:58 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:34:59.711777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36b7562-6e02-4304-874d-71abe65509a4 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:59.713028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36b7562-6e02-4304-874d-71abe65509a4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:59.713344 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36b7562-6e02-4304-874d-71abe65509a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:59.757704 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-d36b7562-6e02-4304-874d-71abe65509a4 None None] MFA Rules not processed for user `00aeab772672403984bdcb8baaa1520e`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:34:59.760894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d36b7562-6e02-4304-874d-71abe65509a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:59.853194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1316/2651] 10.222.0.85 () {58 vars in 967 bytes} [Tue Aug 30 04:34:59 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:34:59.864047 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d2b2e5-ecaa-4011-bb75-6f3babfee4d1 service nova] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:34:59.866298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-38d2b2e5-ecaa-4011-bb75-6f3babfee4d1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:34:59.933031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:34:59.950375 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d2b2e5-ecaa-4011-bb75-6f3babfee4d1 service nova] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:34:59.955557 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6790c06c-eecf-46dd-abdd-8651c7af1b83 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cb539ef6a69406baa70f660e1a815b4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:34:59.956182 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6790c06c-eecf-46dd-abdd-8651c7af1b83 service swift] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:34:59.956462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6790c06c-eecf-46dd-abdd-8651c7af1b83 service swift] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:34:59.956738 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6790c06c-eecf-46dd-abdd-8651c7af1b83 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:34:59.966207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6790c06c-eecf-46dd-abdd-8651c7af1b83 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:34:59.967922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6790c06c-eecf-46dd-abdd-8651c7af1b83 service swift] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:34:59.976681 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1336/2652] 10.222.0.85 () {60 vars in 1384 bytes} [Tue Aug 30 04:34:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 30 04:35:09.843553 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=34 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:09.882232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36b7562-6e02-4304-874d-71abe65509a4 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:09.884720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d36b7562-6e02-4304-874d-71abe65509a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:09.958932 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36b7562-6e02-4304-874d-71abe65509a4 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:09.962452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05885d66-f40c-4a76-868d-166b22750138 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:09.963074 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05885d66-f40c-4a76-868d-166b22750138 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:09.963336 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05885d66-f40c-4a76-868d-166b22750138 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:09.963606 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05885d66-f40c-4a76-868d-166b22750138 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:09.965978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-05885d66-f40c-4a76-868d-166b22750138 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:09.974077 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=6 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:10.036017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05885d66-f40c-4a76-868d-166b22750138 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:10.037550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05885d66-f40c-4a76-868d-166b22750138 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:10.048975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1317/2653] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:35:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:35:13.381267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6790c06c-eecf-46dd-abdd-8651c7af1b83 service swift] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:13.382761 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-6790c06c-eecf-46dd-abdd-8651c7af1b83 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:13.397542 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6790c06c-eecf-46dd-abdd-8651c7af1b83 service swift] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:13.402491 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab2eaf9-487d-4b23-9325-5660f3d56b4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:13.403269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab2eaf9-487d-4b23-9325-5660f3d56b4f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:13.403880 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab2eaf9-487d-4b23-9325-5660f3d56b4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:13.404505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab2eaf9-487d-4b23-9325-5660f3d56b4f admin admin] PATH_INFO: `/v3/users/8ef7a83a87d44fc0a5647cb60057c036` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:13.405682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab2eaf9-487d-4b23-9325-5660f3d56b4f admin admin] RBAC: Authorizing `identity:delete_user(user_id=8ef7a83a87d44fc0a5647cb60057c036)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:13.407650 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab2eaf9-487d-4b23-9325-5660f3d56b4f admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:13.477525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-0ab2eaf9-487d-4b23-9325-5660f3d56b4f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8ef7a83a87d44fc0a5647cb60057c036'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:13.487737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-0ab2eaf9-487d-4b23-9325-5660f3d56b4f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8ef7a83a87d44fc0a5647cb60057c036'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:13.488374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-0ab2eaf9-487d-4b23-9325-5660f3d56b4f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8ef7a83a87d44fc0a5647cb60057c036'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:13.503711 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-0ab2eaf9-487d-4b23-9325-5660f3d56b4f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8ef7a83a87d44fc0a5647cb60057c036'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:13.504538 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-0ab2eaf9-487d-4b23-9325-5660f3d56b4f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8ef7a83a87d44fc0a5647cb60057c036'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:13.519425 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1337/2654] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:35:13 2022] DELETE /identity/v3/users/8ef7a83a87d44fc0a5647cb60057c036 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:13.692601 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05885d66-f40c-4a76-868d-166b22750138 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:13.705805 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05885d66-f40c-4a76-868d-166b22750138 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:13.765400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01e3d22-b270-4b3f-ab8c-47e27f942be9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:13.765963 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01e3d22-b270-4b3f-ab8c-47e27f942be9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:13.766242 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01e3d22-b270-4b3f-ab8c-47e27f942be9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:13.766518 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01e3d22-b270-4b3f-ab8c-47e27f942be9 admin admin] PATH_INFO: `/v3/projects/d7deffddf9934a028bfdb0b1c965c618` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:13.767630 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01e3d22-b270-4b3f-ab8c-47e27f942be9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d7deffddf9934a028bfdb0b1c965c618)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:13.769554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01e3d22-b270-4b3f-ab8c-47e27f942be9 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:13.815558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a01e3d22-b270-4b3f-ab8c-47e27f942be9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd7deffddf9934a028bfdb0b1c965c618', 'request_id': 'req-a01e3d22-b270-4b3f-ab8c-47e27f942be9'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:13.817536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:13.822505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a01e3d22-b270-4b3f-ab8c-47e27f942be9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd7deffddf9934a028bfdb0b1c965c618', 'request_id': 'req-a01e3d22-b270-4b3f-ab8c-47e27f942be9'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:13.830212 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a01e3d22-b270-4b3f-ab8c-47e27f942be9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd7deffddf9934a028bfdb0b1c965c618', 'request_id': 'req-a01e3d22-b270-4b3f-ab8c-47e27f942be9'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:13.841607 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a01e3d22-b270-4b3f-ab8c-47e27f942be9 admin admin] The token cache is being invalidate because project d7deffddf9934a028bfdb0b1c965c618 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:35:13.841894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a01e3d22-b270-4b3f-ab8c-47e27f942be9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:13.842543 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a01e3d22-b270-4b3f-ab8c-47e27f942be9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:13.844033 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1318/2655] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:35:13 2022] DELETE /identity/v3/projects/d7deffddf9934a028bfdb0b1c965c618 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:13.848994 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28c7c6c-9c48-4d2a-8ed3-972572b92228 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:13.849309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28c7c6c-9c48-4d2a-8ed3-972572b92228 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:13.849581 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28c7c6c-9c48-4d2a-8ed3-972572b92228 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:13.869947 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-b28c7c6c-9c48-4d2a-8ed3-972572b92228 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:13.871757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b28c7c6c-9c48-4d2a-8ed3-972572b92228 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:13.931276 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1338/2656] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:35:13.939863 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada52098-9ef8-477d-8025-89a47649879e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:13.940149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada52098-9ef8-477d-8025-89a47649879e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:13.940426 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada52098-9ef8-477d-8025-89a47649879e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:13.960635 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ada52098-9ef8-477d-8025-89a47649879e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:13.961884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ada52098-9ef8-477d-8025-89a47649879e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:13.970050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1319/2657] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:13.974360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28c7c6c-9c48-4d2a-8ed3-972572b92228 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:13.984924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28c7c6c-9c48-4d2a-8ed3-972572b92228 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:13.989509 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e754b83f-fbee-4b92-9e16-a6b2179ef8ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:13.990107 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e754b83f-fbee-4b92-9e16-a6b2179ef8ab admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:13.990401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e754b83f-fbee-4b92-9e16-a6b2179ef8ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:13.990661 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e754b83f-fbee-4b92-9e16-a6b2179ef8ab admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:13.991308 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e754b83f-fbee-4b92-9e16-a6b2179ef8ab admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:35:13.991610 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e754b83f-fbee-4b92-9e16-a6b2179ef8ab admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:13.993252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e754b83f-fbee-4b92-9e16-a6b2179ef8ab admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:14.005468 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1339/2658] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:35:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:35:14.009800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada52098-9ef8-477d-8025-89a47649879e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:14.019007 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada52098-9ef8-477d-8025-89a47649879e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:14.023506 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37918f9-b318-4f1a-bee7-74a6342db944 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:14.024092 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37918f9-b318-4f1a-bee7-74a6342db944 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:14.024366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37918f9-b318-4f1a-bee7-74a6342db944 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:14.024634 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37918f9-b318-4f1a-bee7-74a6342db944 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:14.025290 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37918f9-b318-4f1a-bee7-74a6342db944 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:14.027616 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37918f9-b318-4f1a-bee7-74a6342db944 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:14.054711 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1320/2659] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:35:14 2022] POST /identity/v3/projects => generated 383 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:35:14.059388 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e754b83f-fbee-4b92-9e16-a6b2179ef8ab admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:14.069311 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e754b83f-fbee-4b92-9e16-a6b2179ef8ab admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:14.138491 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5869eae0-0720-4ffe-b4e5-27c79400efd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:14.139116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5869eae0-0720-4ffe-b4e5-27c79400efd8 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:14.139396 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5869eae0-0720-4ffe-b4e5-27c79400efd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:14.139670 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5869eae0-0720-4ffe-b4e5-27c79400efd8 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:14.140317 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5869eae0-0720-4ffe-b4e5-27c79400efd8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:14.142315 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5869eae0-0720-4ffe-b4e5-27c79400efd8 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:14.150490 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:14.166802 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1340/2660] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:35:14 2022] POST /identity/v3/users => generated 292 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:35:14.173883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37918f9-b318-4f1a-bee7-74a6342db944 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:14.184775 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37918f9-b318-4f1a-bee7-74a6342db944 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:14.189612 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76f739c-5780-4a5b-93ab-de4b24c6b46a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:14.198672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76f739c-5780-4a5b-93ab-de4b24c6b46a admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:14.198672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76f739c-5780-4a5b-93ab-de4b24c6b46a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:14.198672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76f739c-5780-4a5b-93ab-de4b24c6b46a admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:14.198672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76f739c-5780-4a5b-93ab-de4b24c6b46a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:14.198672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76f739c-5780-4a5b-93ab-de4b24c6b46a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:14.208356 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1321/2661] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:14 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:35:14.209949 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5869eae0-0720-4ffe-b4e5-27c79400efd8 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:14.220781 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5869eae0-0720-4ffe-b4e5-27c79400efd8 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:14.225749 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff65c8fe-e152-45a8-818d-87e5160922a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:14.226346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff65c8fe-e152-45a8-818d-87e5160922a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:14.226623 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff65c8fe-e152-45a8-818d-87e5160922a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:14.226891 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff65c8fe-e152-45a8-818d-87e5160922a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:14.227530 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff65c8fe-e152-45a8-818d-87e5160922a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:14.229150 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff65c8fe-e152-45a8-818d-87e5160922a4 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:14.243350 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1341/2662] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:14 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:14.247739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76f739c-5780-4a5b-93ab-de4b24c6b46a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:14.258389 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76f739c-5780-4a5b-93ab-de4b24c6b46a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:14.263449 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd569d28-9acb-4191-8bdc-51f9e6bdc12f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:14.263840 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd569d28-9acb-4191-8bdc-51f9e6bdc12f admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:14.264122 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd569d28-9acb-4191-8bdc-51f9e6bdc12f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:14.264401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd569d28-9acb-4191-8bdc-51f9e6bdc12f admin admin] PATH_INFO: `/v3/projects/ce5e96b987d1430e99fd55c317dfd544/users/f01f50b99c5d4899ae94f29630d25913/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:14.298345 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd569d28-9acb-4191-8bdc-51f9e6bdc12f admin admin] RBAC: Authorizing `identity:create_grant(project_id=ce5e96b987d1430e99fd55c317dfd544, user_id=f01f50b99c5d4899ae94f29630d25913, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:14.303993 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd569d28-9acb-4191-8bdc-51f9e6bdc12f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:14.323449 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1322/2663] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:35:14 2022] PUT /identity/v3/projects/ce5e96b987d1430e99fd55c317dfd544/users/f01f50b99c5d4899ae94f29630d25913/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:14.334020 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71057031-27d8-4ae2-981a-e8a7d38dda6d None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:14.334332 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71057031-27d8-4ae2-981a-e8a7d38dda6d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:14.334667 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71057031-27d8-4ae2-981a-e8a7d38dda6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:14.364478 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-71057031-27d8-4ae2-981a-e8a7d38dda6d None None] MFA Rules not processed for user `f01f50b99c5d4899ae94f29630d25913`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:14.365713 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-71057031-27d8-4ae2-981a-e8a7d38dda6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:14.431403 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1342/2664] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:14 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:14.443169 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd569d28-9acb-4191-8bdc-51f9e6bdc12f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:14.444914 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-dd569d28-9acb-4191-8bdc-51f9e6bdc12f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:14.531111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd569d28-9acb-4191-8bdc-51f9e6bdc12f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:14.537104 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6be998-ee5f-486d-b247-e6e503de9d9a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:14.537664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6be998-ee5f-486d-b247-e6e503de9d9a service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:14.537950 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6be998-ee5f-486d-b247-e6e503de9d9a service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:14.538229 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6be998-ee5f-486d-b247-e6e503de9d9a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:14.546704 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a6be998-ee5f-486d-b247-e6e503de9d9a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:14.548315 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a6be998-ee5f-486d-b247-e6e503de9d9a service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:14.557274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1323/2665] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:35:14 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:35:22.664395 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb297f8-b055-4c74-a95b-af3885192df2 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:22.664877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb297f8-b055-4c74-a95b-af3885192df2 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:22.664974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb297f8-b055-4c74-a95b-af3885192df2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:22.684517 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-bcb297f8-b055-4c74-a95b-af3885192df2 None None] MFA Rules not processed for user `2625939a0b9a43d59452df9ba9982ecb`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:22.685751 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-bcb297f8-b055-4c74-a95b-af3885192df2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:22.739217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1343/2666] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:22 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:22.999733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6be998-ee5f-486d-b247-e6e503de9d9a service nova] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:23.063889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6be998-ee5f-486d-b247-e6e503de9d9a service nova] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:23.071383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7767e3d-0fde-4ea9-906a-6cc842c728e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:23.071383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7767e3d-0fde-4ea9-906a-6cc842c728e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:23.071383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7767e3d-0fde-4ea9-906a-6cc842c728e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:23.071383 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7767e3d-0fde-4ea9-906a-6cc842c728e6 admin admin] PATH_INFO: `/v3/users/f01f50b99c5d4899ae94f29630d25913` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:23.072348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7767e3d-0fde-4ea9-906a-6cc842c728e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f01f50b99c5d4899ae94f29630d25913)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:23.074834 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7767e3d-0fde-4ea9-906a-6cc842c728e6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:23.079648 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:23.144079 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e7767e3d-0fde-4ea9-906a-6cc842c728e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f01f50b99c5d4899ae94f29630d25913'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:23.156794 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e7767e3d-0fde-4ea9-906a-6cc842c728e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f01f50b99c5d4899ae94f29630d25913'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:23.163880 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e7767e3d-0fde-4ea9-906a-6cc842c728e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f01f50b99c5d4899ae94f29630d25913'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:23.172583 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e7767e3d-0fde-4ea9-906a-6cc842c728e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f01f50b99c5d4899ae94f29630d25913'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:23.173516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e7767e3d-0fde-4ea9-906a-6cc842c728e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f01f50b99c5d4899ae94f29630d25913'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:23.181182 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1324/2667] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:35:22 2022] DELETE /identity/v3/users/f01f50b99c5d4899ae94f29630d25913 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:23.186444 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25358cc-1127-4386-9dd0-630eba6974cd None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:23.186724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25358cc-1127-4386-9dd0-630eba6974cd None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:23.186998 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25358cc-1127-4386-9dd0-630eba6974cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:23.206965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-e25358cc-1127-4386-9dd0-630eba6974cd None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:23.208207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e25358cc-1127-4386-9dd0-630eba6974cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:23.273021 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1344/2668] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:23.282905 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7767e3d-0fde-4ea9-906a-6cc842c728e6 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:23.284443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e7767e3d-0fde-4ea9-906a-6cc842c728e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:23.360028 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7767e3d-0fde-4ea9-906a-6cc842c728e6 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:23.363791 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629ddcee-8c49-4150-9543-9e07b21da09d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:23.364393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629ddcee-8c49-4150-9543-9e07b21da09d service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:23.364792 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629ddcee-8c49-4150-9543-9e07b21da09d service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:23.365099 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629ddcee-8c49-4150-9543-9e07b21da09d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:23.377909 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629ddcee-8c49-4150-9543-9e07b21da09d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:23.379399 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629ddcee-8c49-4150-9543-9e07b21da09d service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:23.390118 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1325/2669] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:35:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:35:23.702565 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25358cc-1127-4386-9dd0-630eba6974cd None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:23.704915 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e25358cc-1127-4386-9dd0-630eba6974cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:23.719201 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25358cc-1127-4386-9dd0-630eba6974cd None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:23.726519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6129f02f-e24c-4713-8ac5-e231f4f20901 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:23.727036 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6129f02f-e24c-4713-8ac5-e231f4f20901 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:23.727314 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6129f02f-e24c-4713-8ac5-e231f4f20901 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:23.727592 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6129f02f-e24c-4713-8ac5-e231f4f20901 admin admin] PATH_INFO: `/v3/projects/ce5e96b987d1430e99fd55c317dfd544` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:23.729061 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6129f02f-e24c-4713-8ac5-e231f4f20901 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ce5e96b987d1430e99fd55c317dfd544)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:23.731541 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6129f02f-e24c-4713-8ac5-e231f4f20901 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:23.741371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:23.799152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-6129f02f-e24c-4713-8ac5-e231f4f20901 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ce5e96b987d1430e99fd55c317dfd544', 'request_id': 'req-6129f02f-e24c-4713-8ac5-e231f4f20901'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:23.805569 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-6129f02f-e24c-4713-8ac5-e231f4f20901 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ce5e96b987d1430e99fd55c317dfd544', 'request_id': 'req-6129f02f-e24c-4713-8ac5-e231f4f20901'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:23.814457 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-6129f02f-e24c-4713-8ac5-e231f4f20901 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ce5e96b987d1430e99fd55c317dfd544', 'request_id': 'req-6129f02f-e24c-4713-8ac5-e231f4f20901'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:23.826813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-6129f02f-e24c-4713-8ac5-e231f4f20901 admin admin] The token cache is being invalidate because project ce5e96b987d1430e99fd55c317dfd544 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:35:23.827057 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-6129f02f-e24c-4713-8ac5-e231f4f20901 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:23.827749 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-6129f02f-e24c-4713-8ac5-e231f4f20901 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:23.829346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1345/2670] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:35:23 2022] DELETE /identity/v3/projects/ce5e96b987d1430e99fd55c317dfd544 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:23.834439 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc69844-a33e-40c2-853e-f8a431003232 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:23.834727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc69844-a33e-40c2-853e-f8a431003232 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:23.834998 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc69844-a33e-40c2-853e-f8a431003232 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:23.856295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-6cc69844-a33e-40c2-853e-f8a431003232 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:23.857560 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6cc69844-a33e-40c2-853e-f8a431003232 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:23.936545 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1326/2671] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:23.944526 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67e3a06-4a5f-4920-9e8a-776c2d847123 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:23.944840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67e3a06-4a5f-4920-9e8a-776c2d847123 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:23.945149 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67e3a06-4a5f-4920-9e8a-776c2d847123 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:23.966669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-b67e3a06-4a5f-4920-9e8a-776c2d847123 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:23.968301 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b67e3a06-4a5f-4920-9e8a-776c2d847123 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:23.984760 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1346/2672] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:23.990825 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc69844-a33e-40c2-853e-f8a431003232 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:24.005476 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc69844-a33e-40c2-853e-f8a431003232 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:24.012020 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79934aac-53d1-4c94-859d-8994f900d59d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:24.012906 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79934aac-53d1-4c94-859d-8994f900d59d admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:24.013341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79934aac-53d1-4c94-859d-8994f900d59d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:24.013749 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79934aac-53d1-4c94-859d-8994f900d59d admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:24.014743 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79934aac-53d1-4c94-859d-8994f900d59d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:35:24.015168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79934aac-53d1-4c94-859d-8994f900d59d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:24.017566 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79934aac-53d1-4c94-859d-8994f900d59d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:24.046544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1327/2673] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:35:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:35:24.047720 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67e3a06-4a5f-4920-9e8a-776c2d847123 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:24.058485 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67e3a06-4a5f-4920-9e8a-776c2d847123 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:24.063796 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492e8374-cc7f-46b5-a80f-386c67ee9741 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:24.064398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492e8374-cc7f-46b5-a80f-386c67ee9741 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:24.064689 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492e8374-cc7f-46b5-a80f-386c67ee9741 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:24.064953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492e8374-cc7f-46b5-a80f-386c67ee9741 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:24.065640 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492e8374-cc7f-46b5-a80f-386c67ee9741 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:24.067549 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492e8374-cc7f-46b5-a80f-386c67ee9741 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:24.109594 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1347/2674] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:35:24 2022] POST /identity/v3/projects => generated 363 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:35:24.110876 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79934aac-53d1-4c94-859d-8994f900d59d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:24.127010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79934aac-53d1-4c94-859d-8994f900d59d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:24.136465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:24.203862 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc8d4fd-7c4a-4b27-ab34-62a51df5ea80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:24.204481 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc8d4fd-7c4a-4b27-ab34-62a51df5ea80 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:24.204766 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc8d4fd-7c4a-4b27-ab34-62a51df5ea80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:24.205062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc8d4fd-7c4a-4b27-ab34-62a51df5ea80 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:24.208542 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc8d4fd-7c4a-4b27-ab34-62a51df5ea80 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:24.211523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc8d4fd-7c4a-4b27-ab34-62a51df5ea80 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:24.243235 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1328/2675] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:35:24 2022] POST /identity/v3/users => generated 282 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:35:24.247789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492e8374-cc7f-46b5-a80f-386c67ee9741 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:24.264693 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492e8374-cc7f-46b5-a80f-386c67ee9741 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:24.269799 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e7beec-146e-4b4f-93fc-8799af3f979b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:24.270407 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e7beec-146e-4b4f-93fc-8799af3f979b admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:24.270685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e7beec-146e-4b4f-93fc-8799af3f979b admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:24.270959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e7beec-146e-4b4f-93fc-8799af3f979b admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:24.271598 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e7beec-146e-4b4f-93fc-8799af3f979b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:24.273234 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e7beec-146e-4b4f-93fc-8799af3f979b admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:24.287664 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1348/2676] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:24 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:24.289060 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc8d4fd-7c4a-4b27-ab34-62a51df5ea80 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:24.300115 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc8d4fd-7c4a-4b27-ab34-62a51df5ea80 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:24.304708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527dd43b-3240-47c3-803b-78d5f58b3580 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:24.305341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527dd43b-3240-47c3-803b-78d5f58b3580 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:24.305638 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527dd43b-3240-47c3-803b-78d5f58b3580 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:24.305887 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527dd43b-3240-47c3-803b-78d5f58b3580 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:24.306530 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527dd43b-3240-47c3-803b-78d5f58b3580 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:24.308146 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527dd43b-3240-47c3-803b-78d5f58b3580 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:24.319091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1329/2677] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:24 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:24.323815 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e7beec-146e-4b4f-93fc-8799af3f979b admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:24.334637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e7beec-146e-4b4f-93fc-8799af3f979b admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:24.340913 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a912ef2-e1b9-4524-bef5-ea4c20007cd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:24.341703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a912ef2-e1b9-4524-bef5-ea4c20007cd0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:24.342053 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a912ef2-e1b9-4524-bef5-ea4c20007cd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:24.342394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a912ef2-e1b9-4524-bef5-ea4c20007cd0 admin admin] PATH_INFO: `/v3/projects/5ea4ff33b4034170a7343def85c23a85/users/ff3ced7cf9354858b3c528277190fb1b/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:24.358040 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a912ef2-e1b9-4524-bef5-ea4c20007cd0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ea4ff33b4034170a7343def85c23a85, user_id=ff3ced7cf9354858b3c528277190fb1b, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:24.360944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a912ef2-e1b9-4524-bef5-ea4c20007cd0 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:24.375210 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1349/2678] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:35:24 2022] PUT /identity/v3/projects/5ea4ff33b4034170a7343def85c23a85/users/ff3ced7cf9354858b3c528277190fb1b/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:24.389287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ae1505-f51e-4411-a09f-348f1fc4d230 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:24.389777 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ae1505-f51e-4411-a09f-348f1fc4d230 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:24.390129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ae1505-f51e-4411-a09f-348f1fc4d230 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:24.420661 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-00ae1505-f51e-4411-a09f-348f1fc4d230 None None] MFA Rules not processed for user `ff3ced7cf9354858b3c528277190fb1b`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:24.422147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-00ae1505-f51e-4411-a09f-348f1fc4d230 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:24.488937 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1330/2679] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:24 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:24.493467 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a912ef2-e1b9-4524-bef5-ea4c20007cd0 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:24.556181 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a912ef2-e1b9-4524-bef5-ea4c20007cd0 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:24.560829 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf2ef5c-3050-4ad7-bcee-a315583286e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:24.561457 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf2ef5c-3050-4ad7-bcee-a315583286e8 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:24.561738 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf2ef5c-3050-4ad7-bcee-a315583286e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:24.562009 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf2ef5c-3050-4ad7-bcee-a315583286e8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:24.562639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf2ef5c-3050-4ad7-bcee-a315583286e8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:24.564595 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf2ef5c-3050-4ad7-bcee-a315583286e8 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:24.591782 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1350/2680] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:35:24 2022] POST /identity/v3/projects => generated 363 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:35:24.595974 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ae1505-f51e-4411-a09f-348f1fc4d230 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:24.606491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ae1505-f51e-4411-a09f-348f1fc4d230 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:24.667927 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3afa87-671c-4bd7-8c6f-ae95250d0636 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:24.668540 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3afa87-671c-4bd7-8c6f-ae95250d0636 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:24.668822 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3afa87-671c-4bd7-8c6f-ae95250d0636 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:24.669126 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3afa87-671c-4bd7-8c6f-ae95250d0636 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:24.669768 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3afa87-671c-4bd7-8c6f-ae95250d0636 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:24.672031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3afa87-671c-4bd7-8c6f-ae95250d0636 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:24.697649 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1331/2681] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:35:24 2022] POST /identity/v3/users => generated 288 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:35:24.699543 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf2ef5c-3050-4ad7-bcee-a315583286e8 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:24.710402 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf2ef5c-3050-4ad7-bcee-a315583286e8 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:24.715317 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204e5145-7178-4832-b280-acbe020f5e84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:24.715893 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204e5145-7178-4832-b280-acbe020f5e84 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:24.716175 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204e5145-7178-4832-b280-acbe020f5e84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:24.716497 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204e5145-7178-4832-b280-acbe020f5e84 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:24.717303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204e5145-7178-4832-b280-acbe020f5e84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:24.718998 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204e5145-7178-4832-b280-acbe020f5e84 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:24.729979 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1351/2682] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:24 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:24.734468 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3afa87-671c-4bd7-8c6f-ae95250d0636 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:24.745942 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3afa87-671c-4bd7-8c6f-ae95250d0636 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:24.750706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330ae1e1-f1b9-429c-ab34-a2c9191ed07c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:24.751358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330ae1e1-f1b9-429c-ab34-a2c9191ed07c admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:24.751642 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330ae1e1-f1b9-429c-ab34-a2c9191ed07c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:24.751919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330ae1e1-f1b9-429c-ab34-a2c9191ed07c admin admin] PATH_INFO: `/v3/projects/5e0d683ca86c4cdf9959928bad7749ff/users/3c10d30acf16491e979c542bad6b0ec4/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:24.754987 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:24.769573 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330ae1e1-f1b9-429c-ab34-a2c9191ed07c admin admin] RBAC: Authorizing `identity:create_grant(project_id=5e0d683ca86c4cdf9959928bad7749ff, user_id=3c10d30acf16491e979c542bad6b0ec4, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:24.772348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330ae1e1-f1b9-429c-ab34-a2c9191ed07c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:24.788568 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1332/2683] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:35:24 2022] PUT /identity/v3/projects/5e0d683ca86c4cdf9959928bad7749ff/users/3c10d30acf16491e979c542bad6b0ec4/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:24.793381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204e5145-7178-4832-b280-acbe020f5e84 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:24.810740 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204e5145-7178-4832-b280-acbe020f5e84 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:24.815066 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:24.879516 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f9a7e0-4ed8-4e50-b826-e5aadbc6b02b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:24.880472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f9a7e0-4ed8-4e50-b826-e5aadbc6b02b admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:24.880761 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f9a7e0-4ed8-4e50-b826-e5aadbc6b02b admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:24.881386 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f9a7e0-4ed8-4e50-b826-e5aadbc6b02b admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:24.882075 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f9a7e0-4ed8-4e50-b826-e5aadbc6b02b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:24.884027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f9a7e0-4ed8-4e50-b826-e5aadbc6b02b admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:24.895992 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1352/2684] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:24 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:35:24.900454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330ae1e1-f1b9-429c-ab34-a2c9191ed07c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:24.911307 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330ae1e1-f1b9-429c-ab34-a2c9191ed07c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:24.916877 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee3461d-0340-47dd-a84a-2ed39da27126 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:24.917668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee3461d-0340-47dd-a84a-2ed39da27126 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:24.918003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee3461d-0340-47dd-a84a-2ed39da27126 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:24.918287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee3461d-0340-47dd-a84a-2ed39da27126 admin admin] PATH_INFO: `/v3/projects/5e0d683ca86c4cdf9959928bad7749ff/users/3c10d30acf16491e979c542bad6b0ec4/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:24.920334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee3461d-0340-47dd-a84a-2ed39da27126 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5e0d683ca86c4cdf9959928bad7749ff, user_id=3c10d30acf16491e979c542bad6b0ec4, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:24.923112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee3461d-0340-47dd-a84a-2ed39da27126 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:24.937016 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1333/2685] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:35:24 2022] PUT /identity/v3/projects/5e0d683ca86c4cdf9959928bad7749ff/users/3c10d30acf16491e979c542bad6b0ec4/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:24.948965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3473d1-d5e3-4ca1-93fd-c97da847bff1 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:24.950476 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3473d1-d5e3-4ca1-93fd-c97da847bff1 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:24.950476 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3473d1-d5e3-4ca1-93fd-c97da847bff1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:24.985624 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-9a3473d1-d5e3-4ca1-93fd-c97da847bff1 None None] MFA Rules not processed for user `3c10d30acf16491e979c542bad6b0ec4`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:24.987053 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-9a3473d1-d5e3-4ca1-93fd-c97da847bff1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:25.063753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1353/2686] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:24 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:35:25.078280 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee3461d-0340-47dd-a84a-2ed39da27126 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:25.080299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bee3461d-0340-47dd-a84a-2ed39da27126 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:25.196149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee3461d-0340-47dd-a84a-2ed39da27126 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:25.199971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53955d89-a86f-48e9-82b6-31a8eafd8327 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:25.200618 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53955d89-a86f-48e9-82b6-31a8eafd8327 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:25.200902 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53955d89-a86f-48e9-82b6-31a8eafd8327 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:25.201207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53955d89-a86f-48e9-82b6-31a8eafd8327 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:25.212226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53955d89-a86f-48e9-82b6-31a8eafd8327 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:25.213659 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53955d89-a86f-48e9-82b6-31a8eafd8327 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:25.224363 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1334/2687] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:35:25 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:35:25.661142 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3473d1-d5e3-4ca1-93fd-c97da847bff1 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:25.733058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3473d1-d5e3-4ca1-93fd-c97da847bff1 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:25.739381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6f6ee7-4ac8-485a-8a29-09a7a65277b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:25.740161 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6f6ee7-4ac8-485a-8a29-09a7a65277b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:25.740518 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6f6ee7-4ac8-485a-8a29-09a7a65277b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:25.740921 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6f6ee7-4ac8-485a-8a29-09a7a65277b3 admin admin] PATH_INFO: `/v3/users/ff3ced7cf9354858b3c528277190fb1b` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:25.742201 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6f6ee7-4ac8-485a-8a29-09a7a65277b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ff3ced7cf9354858b3c528277190fb1b)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:25.744588 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6f6ee7-4ac8-485a-8a29-09a7a65277b3 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:25.811703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:25.833473 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ee6f6ee7-4ac8-485a-8a29-09a7a65277b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ff3ced7cf9354858b3c528277190fb1b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:25.843539 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ee6f6ee7-4ac8-485a-8a29-09a7a65277b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ff3ced7cf9354858b3c528277190fb1b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:25.844699 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ee6f6ee7-4ac8-485a-8a29-09a7a65277b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ff3ced7cf9354858b3c528277190fb1b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:25.862409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ee6f6ee7-4ac8-485a-8a29-09a7a65277b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ff3ced7cf9354858b3c528277190fb1b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:25.863376 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ee6f6ee7-4ac8-485a-8a29-09a7a65277b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ff3ced7cf9354858b3c528277190fb1b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:25.877676 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1354/2688] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:35:25 2022] DELETE /identity/v3/users/ff3ced7cf9354858b3c528277190fb1b => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:25.882151 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019a0072-f366-4f3e-ab7f-b3de0feb25b0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:25.882516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019a0072-f366-4f3e-ab7f-b3de0feb25b0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:25.882854 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019a0072-f366-4f3e-ab7f-b3de0feb25b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:25.910345 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-019a0072-f366-4f3e-ab7f-b3de0feb25b0 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:25.912027 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-019a0072-f366-4f3e-ab7f-b3de0feb25b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:25.989323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1335/2689] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:35:26.003957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6f6ee7-4ac8-485a-8a29-09a7a65277b3 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:26.005460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ee6f6ee7-4ac8-485a-8a29-09a7a65277b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:26.075566 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6f6ee7-4ac8-485a-8a29-09a7a65277b3 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:26.079925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067a80e9-f312-44ab-abb3-9479665f1fca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:26.080541 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067a80e9-f312-44ab-abb3-9479665f1fca service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:26.080816 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067a80e9-f312-44ab-abb3-9479665f1fca service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:26.081100 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067a80e9-f312-44ab-abb3-9479665f1fca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:26.091665 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067a80e9-f312-44ab-abb3-9479665f1fca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:26.093148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067a80e9-f312-44ab-abb3-9479665f1fca service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:26.103500 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1355/2690] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:35:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:35:26.427037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019a0072-f366-4f3e-ab7f-b3de0feb25b0 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:26.430540 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-019a0072-f366-4f3e-ab7f-b3de0feb25b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:26.462226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019a0072-f366-4f3e-ab7f-b3de0feb25b0 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:26.470587 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a3a208-c4a8-489b-901c-609599a7c401 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:26.471592 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a3a208-c4a8-489b-901c-609599a7c401 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:26.472066 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a3a208-c4a8-489b-901c-609599a7c401 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:26.472536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a3a208-c4a8-489b-901c-609599a7c401 admin admin] PATH_INFO: `/v3/projects/5ea4ff33b4034170a7343def85c23a85` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:26.474605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a3a208-c4a8-489b-901c-609599a7c401 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5ea4ff33b4034170a7343def85c23a85)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:26.478189 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a3a208-c4a8-489b-901c-609599a7c401 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:26.544368 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f1a3a208-c4a8-489b-901c-609599a7c401 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5ea4ff33b4034170a7343def85c23a85', 'request_id': 'req-f1a3a208-c4a8-489b-901c-609599a7c401'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:26.545663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=94,keystone:INSERT=2,keystone:DELETE=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:26.551548 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f1a3a208-c4a8-489b-901c-609599a7c401 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5ea4ff33b4034170a7343def85c23a85', 'request_id': 'req-f1a3a208-c4a8-489b-901c-609599a7c401'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:26.561966 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f1a3a208-c4a8-489b-901c-609599a7c401 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5ea4ff33b4034170a7343def85c23a85', 'request_id': 'req-f1a3a208-c4a8-489b-901c-609599a7c401'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:26.575277 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f1a3a208-c4a8-489b-901c-609599a7c401 admin admin] The token cache is being invalidate because project 5ea4ff33b4034170a7343def85c23a85 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:35:26.575609 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f1a3a208-c4a8-489b-901c-609599a7c401 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:26.576247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f1a3a208-c4a8-489b-901c-609599a7c401 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:26.584236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1336/2691] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:35:26 2022] DELETE /identity/v3/projects/5ea4ff33b4034170a7343def85c23a85 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:26.585525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067a80e9-f312-44ab-abb3-9479665f1fca service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:26.611401 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067a80e9-f312-44ab-abb3-9479665f1fca service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:26.704811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5123aed-438d-4367-8511-b4a433ccfe79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:26.705515 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5123aed-438d-4367-8511-b4a433ccfe79 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:26.705802 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5123aed-438d-4367-8511-b4a433ccfe79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:26.706126 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5123aed-438d-4367-8511-b4a433ccfe79 admin admin] PATH_INFO: `/v3/users/3c10d30acf16491e979c542bad6b0ec4` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:26.707343 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5123aed-438d-4367-8511-b4a433ccfe79 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3c10d30acf16491e979c542bad6b0ec4)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:26.709375 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5123aed-438d-4367-8511-b4a433ccfe79 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:26.788801 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a5123aed-438d-4367-8511-b4a433ccfe79 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3c10d30acf16491e979c542bad6b0ec4'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:26.803062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:26.811526 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a5123aed-438d-4367-8511-b4a433ccfe79 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3c10d30acf16491e979c542bad6b0ec4'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:26.812987 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a5123aed-438d-4367-8511-b4a433ccfe79 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3c10d30acf16491e979c542bad6b0ec4'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:26.844965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a5123aed-438d-4367-8511-b4a433ccfe79 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3c10d30acf16491e979c542bad6b0ec4'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:26.846199 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a5123aed-438d-4367-8511-b4a433ccfe79 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3c10d30acf16491e979c542bad6b0ec4'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:26.872634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1356/2692] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:35:26 2022] DELETE /identity/v3/users/3c10d30acf16491e979c542bad6b0ec4 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:27.396240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a3a208-c4a8-489b-901c-609599a7c401 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:27.398077 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f1a3a208-c4a8-489b-901c-609599a7c401 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:27.413963 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a3a208-c4a8-489b-901c-609599a7c401 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:27.486892 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f36653-d775-4c1d-98a7-5a658f403f91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:27.487540 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f36653-d775-4c1d-98a7-5a658f403f91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:27.487824 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f36653-d775-4c1d-98a7-5a658f403f91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:27.488101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f36653-d775-4c1d-98a7-5a658f403f91 admin admin] PATH_INFO: `/v3/projects/5e0d683ca86c4cdf9959928bad7749ff` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:27.489278 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f36653-d775-4c1d-98a7-5a658f403f91 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5e0d683ca86c4cdf9959928bad7749ff)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:27.491220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f36653-d775-4c1d-98a7-5a658f403f91 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:27.541030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c8f36653-d775-4c1d-98a7-5a658f403f91 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5e0d683ca86c4cdf9959928bad7749ff', 'request_id': 'req-c8f36653-d775-4c1d-98a7-5a658f403f91'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:27.547523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c8f36653-d775-4c1d-98a7-5a658f403f91 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5e0d683ca86c4cdf9959928bad7749ff', 'request_id': 'req-c8f36653-d775-4c1d-98a7-5a658f403f91'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:27.555296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c8f36653-d775-4c1d-98a7-5a658f403f91 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5e0d683ca86c4cdf9959928bad7749ff', 'request_id': 'req-c8f36653-d775-4c1d-98a7-5a658f403f91'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:27.567855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c8f36653-d775-4c1d-98a7-5a658f403f91 admin admin] The token cache is being invalidate because project 5e0d683ca86c4cdf9959928bad7749ff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:35:27.568142 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c8f36653-d775-4c1d-98a7-5a658f403f91 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:27.568796 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c8f36653-d775-4c1d-98a7-5a658f403f91 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:27.570688 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1337/2693] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:35:27 2022] DELETE /identity/v3/projects/5e0d683ca86c4cdf9959928bad7749ff => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:27.576206 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d14022-c2e4-4e82-a3e6-ec00ca9900cc None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:27.576580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d14022-c2e4-4e82-a3e6-ec00ca9900cc None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:27.576860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d14022-c2e4-4e82-a3e6-ec00ca9900cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:27.596888 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-c4d14022-c2e4-4e82-a3e6-ec00ca9900cc None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:27.598386 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c4d14022-c2e4-4e82-a3e6-ec00ca9900cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:27.661837 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1357/2694] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:27.670136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249e819b-ba4b-4573-af7f-8ac40c419259 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:27.670458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249e819b-ba4b-4573-af7f-8ac40c419259 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:27.670738 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249e819b-ba4b-4573-af7f-8ac40c419259 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:27.695244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-249e819b-ba4b-4573-af7f-8ac40c419259 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:27.696487 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-249e819b-ba4b-4573-af7f-8ac40c419259 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:27.705102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1338/2695] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:27.709784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d14022-c2e4-4e82-a3e6-ec00ca9900cc None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:27.719890 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d14022-c2e4-4e82-a3e6-ec00ca9900cc None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:27.724468 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d09f97-ea55-4442-887e-032a4fd6daa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:27.725081 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d09f97-ea55-4442-887e-032a4fd6daa5 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:27.725363 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d09f97-ea55-4442-887e-032a4fd6daa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:27.725634 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d09f97-ea55-4442-887e-032a4fd6daa5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:27.726307 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d09f97-ea55-4442-887e-032a4fd6daa5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:35:27.726606 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d09f97-ea55-4442-887e-032a4fd6daa5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:27.728217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d09f97-ea55-4442-887e-032a4fd6daa5 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:27.744236 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1358/2696] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:35:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:35:27.749140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249e819b-ba4b-4573-af7f-8ac40c419259 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:27.758444 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249e819b-ba4b-4573-af7f-8ac40c419259 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:27.765217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd7b7a5-5f8a-4c19-9395-c1c89dc3efb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:27.765832 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd7b7a5-5f8a-4c19-9395-c1c89dc3efb1 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:27.766108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd7b7a5-5f8a-4c19-9395-c1c89dc3efb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:27.766404 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd7b7a5-5f8a-4c19-9395-c1c89dc3efb1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:27.767078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd7b7a5-5f8a-4c19-9395-c1c89dc3efb1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:27.768971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd7b7a5-5f8a-4c19-9395-c1c89dc3efb1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:27.798423 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1339/2697] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:35:27 2022] POST /identity/v3/projects => generated 363 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:35:27.802677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d09f97-ea55-4442-887e-032a4fd6daa5 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:27.813500 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d09f97-ea55-4442-887e-032a4fd6daa5 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:27.872693 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d434cc5d-66f6-4935-a533-5fc27aee0174 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:27.873340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d434cc5d-66f6-4935-a533-5fc27aee0174 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:27.873619 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d434cc5d-66f6-4935-a533-5fc27aee0174 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:27.873890 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d434cc5d-66f6-4935-a533-5fc27aee0174 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:27.874525 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d434cc5d-66f6-4935-a533-5fc27aee0174 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:27.876442 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d434cc5d-66f6-4935-a533-5fc27aee0174 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:27.909112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1359/2698] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:35:27 2022] POST /identity/v3/users => generated 282 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:35:27.913326 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd7b7a5-5f8a-4c19-9395-c1c89dc3efb1 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:27.924486 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd7b7a5-5f8a-4c19-9395-c1c89dc3efb1 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:27.929195 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db459dc5-3dd0-4ec4-b0e3-dc0523aef343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:27.929856 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db459dc5-3dd0-4ec4-b0e3-dc0523aef343 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:27.930138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db459dc5-3dd0-4ec4-b0e3-dc0523aef343 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:27.930419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db459dc5-3dd0-4ec4-b0e3-dc0523aef343 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:27.931065 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db459dc5-3dd0-4ec4-b0e3-dc0523aef343 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:27.933265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db459dc5-3dd0-4ec4-b0e3-dc0523aef343 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:27.944361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1340/2699] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:27 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:27.949095 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d434cc5d-66f6-4935-a533-5fc27aee0174 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:27.959337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d434cc5d-66f6-4935-a533-5fc27aee0174 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:27.964108 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726501d9-3e2d-4395-a4b7-a0b4ed350481 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:27.964711 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726501d9-3e2d-4395-a4b7-a0b4ed350481 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:27.964993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726501d9-3e2d-4395-a4b7-a0b4ed350481 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:27.965306 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726501d9-3e2d-4395-a4b7-a0b4ed350481 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:27.965977 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726501d9-3e2d-4395-a4b7-a0b4ed350481 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:27.967566 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726501d9-3e2d-4395-a4b7-a0b4ed350481 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:27.980076 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1360/2700] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:27 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:27.984743 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db459dc5-3dd0-4ec4-b0e3-dc0523aef343 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:27.995689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db459dc5-3dd0-4ec4-b0e3-dc0523aef343 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:28.000939 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1027b34e-25f7-4a92-b125-e0685f680155 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:28.001647 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1027b34e-25f7-4a92-b125-e0685f680155 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:28.001931 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1027b34e-25f7-4a92-b125-e0685f680155 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:28.002206 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1027b34e-25f7-4a92-b125-e0685f680155 admin admin] PATH_INFO: `/v3/projects/043212768c2d44a283ab1eeabc295ba7/users/89f3ae28f5d145979d2d2403e50c8807/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:28.017810 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1027b34e-25f7-4a92-b125-e0685f680155 admin admin] RBAC: Authorizing `identity:create_grant(project_id=043212768c2d44a283ab1eeabc295ba7, user_id=89f3ae28f5d145979d2d2403e50c8807, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:28.020493 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1027b34e-25f7-4a92-b125-e0685f680155 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:28.033857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1341/2701] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:35:27 2022] PUT /identity/v3/projects/043212768c2d44a283ab1eeabc295ba7/users/89f3ae28f5d145979d2d2403e50c8807/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:28.046321 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb6ddd8-39d7-4a89-9590-5adcffe01d66 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:28.046708 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb6ddd8-39d7-4a89-9590-5adcffe01d66 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:28.046990 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb6ddd8-39d7-4a89-9590-5adcffe01d66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:28.075496 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-7fb6ddd8-39d7-4a89-9590-5adcffe01d66 None None] MFA Rules not processed for user `89f3ae28f5d145979d2d2403e50c8807`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:28.076729 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-7fb6ddd8-39d7-4a89-9590-5adcffe01d66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:28.101391 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:28.152316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1361/2702] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:28 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:28.158149 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1027b34e-25f7-4a92-b125-e0685f680155 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:28.162671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:DELETE=10,keystone:SELECT=86,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:28.248268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1027b34e-25f7-4a92-b125-e0685f680155 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:28.253185 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc315469-ef4d-4fbd-939f-7fc55de0bfc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:28.253809 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc315469-ef4d-4fbd-939f-7fc55de0bfc1 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:28.254086 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc315469-ef4d-4fbd-939f-7fc55de0bfc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:28.254356 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc315469-ef4d-4fbd-939f-7fc55de0bfc1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:28.255001 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc315469-ef4d-4fbd-939f-7fc55de0bfc1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:28.256920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc315469-ef4d-4fbd-939f-7fc55de0bfc1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:28.290589 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1342/2703] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:35:28 2022] POST /identity/v3/projects => generated 365 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:35:28.301394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb6ddd8-39d7-4a89-9590-5adcffe01d66 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:28.314521 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb6ddd8-39d7-4a89-9590-5adcffe01d66 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:28.402732 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a7a6dc-88ec-45ce-9d0b-1a7d4844b79b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:28.403424 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a7a6dc-88ec-45ce-9d0b-1a7d4844b79b admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:28.403619 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a7a6dc-88ec-45ce-9d0b-1a7d4844b79b admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:28.403957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a7a6dc-88ec-45ce-9d0b-1a7d4844b79b admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:28.404906 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a7a6dc-88ec-45ce-9d0b-1a7d4844b79b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:28.406882 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a7a6dc-88ec-45ce-9d0b-1a7d4844b79b admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:28.432415 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1362/2704] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:35:28 2022] POST /identity/v3/users => generated 289 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:35:28.434182 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc315469-ef4d-4fbd-939f-7fc55de0bfc1 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:28.444733 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc315469-ef4d-4fbd-939f-7fc55de0bfc1 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:28.449328 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7939031-2906-4156-ba23-75370c8883ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:28.449926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7939031-2906-4156-ba23-75370c8883ad admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:28.450204 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7939031-2906-4156-ba23-75370c8883ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:28.450472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7939031-2906-4156-ba23-75370c8883ad admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:28.451305 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7939031-2906-4156-ba23-75370c8883ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:28.452897 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7939031-2906-4156-ba23-75370c8883ad admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:28.464836 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1343/2705] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:28.469271 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a7a6dc-88ec-45ce-9d0b-1a7d4844b79b admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:28.480330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a7a6dc-88ec-45ce-9d0b-1a7d4844b79b admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:28.489139 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be019f8a-069f-413c-950c-20a78a47211d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:28.489877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be019f8a-069f-413c-950c-20a78a47211d admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:28.490218 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be019f8a-069f-413c-950c-20a78a47211d admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:28.490555 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be019f8a-069f-413c-950c-20a78a47211d admin admin] PATH_INFO: `/v3/projects/6f1174a665b248e09e29358cecdd048f/users/faf34a5ebdcc4d959264c4649ffae5de/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:28.510523 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be019f8a-069f-413c-950c-20a78a47211d admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f1174a665b248e09e29358cecdd048f, user_id=faf34a5ebdcc4d959264c4649ffae5de, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:28.512586 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be019f8a-069f-413c-950c-20a78a47211d admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:28.529876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1363/2706] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:35:28 2022] PUT /identity/v3/projects/6f1174a665b248e09e29358cecdd048f/users/faf34a5ebdcc4d959264c4649ffae5de/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:35:28.534710 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7939031-2906-4156-ba23-75370c8883ad admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:28.548859 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7939031-2906-4156-ba23-75370c8883ad admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:28.607397 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8296dd48-436a-4253-9c9b-bacbddf8bb81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:28.608040 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8296dd48-436a-4253-9c9b-bacbddf8bb81 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:28.608322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8296dd48-436a-4253-9c9b-bacbddf8bb81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:28.608598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8296dd48-436a-4253-9c9b-bacbddf8bb81 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:28.609564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8296dd48-436a-4253-9c9b-bacbddf8bb81 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:28.611166 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8296dd48-436a-4253-9c9b-bacbddf8bb81 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:28.621761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1344/2707] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:28 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:35:28.627107 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be019f8a-069f-413c-950c-20a78a47211d admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:28.643696 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be019f8a-069f-413c-950c-20a78a47211d admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:28.651651 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e28670-ec8d-49bd-a4ee-0d44bd7dd028 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:28.652580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e28670-ec8d-49bd-a4ee-0d44bd7dd028 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:28.652983 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e28670-ec8d-49bd-a4ee-0d44bd7dd028 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:28.653417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e28670-ec8d-49bd-a4ee-0d44bd7dd028 admin admin] PATH_INFO: `/v3/projects/6f1174a665b248e09e29358cecdd048f/users/faf34a5ebdcc4d959264c4649ffae5de/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:28.656273 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e28670-ec8d-49bd-a4ee-0d44bd7dd028 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f1174a665b248e09e29358cecdd048f, user_id=faf34a5ebdcc4d959264c4649ffae5de, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:28.660594 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e28670-ec8d-49bd-a4ee-0d44bd7dd028 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:28.675918 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1364/2708] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:35:28 2022] PUT /identity/v3/projects/6f1174a665b248e09e29358cecdd048f/users/faf34a5ebdcc4d959264c4649ffae5de/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:28.687315 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3b093b-c4f5-4082-96ec-ca112570906a None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:28.687595 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3b093b-c4f5-4082-96ec-ca112570906a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:28.687874 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3b093b-c4f5-4082-96ec-ca112570906a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:28.717336 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-8c3b093b-c4f5-4082-96ec-ca112570906a None None] MFA Rules not processed for user `faf34a5ebdcc4d959264c4649ffae5de`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:28.718564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8c3b093b-c4f5-4082-96ec-ca112570906a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:28.795568 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:28.801646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1345/2709] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:28 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:28.814442 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e28670-ec8d-49bd-a4ee-0d44bd7dd028 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:28.816258 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-25e28670-ec8d-49bd-a4ee-0d44bd7dd028 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:28.875912 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e28670-ec8d-49bd-a4ee-0d44bd7dd028 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:28.879792 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f77b78-dc5e-417e-89d0-ebce450a0a96 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:28.880390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f77b78-dc5e-417e-89d0-ebce450a0a96 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:28.880663 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f77b78-dc5e-417e-89d0-ebce450a0a96 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:28.880930 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f77b78-dc5e-417e-89d0-ebce450a0a96 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:28.890847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f77b78-dc5e-417e-89d0-ebce450a0a96 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:28.892263 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f77b78-dc5e-417e-89d0-ebce450a0a96 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:28.903258 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1365/2710] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:35:28 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:35:28.986103 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3b093b-c4f5-4082-96ec-ca112570906a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:28.997095 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3b093b-c4f5-4082-96ec-ca112570906a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:29.000684 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d999cfac-10cf-48ea-bd09-ca9ef65a68ae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:29.001390 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d999cfac-10cf-48ea-bd09-ca9ef65a68ae service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:29.001701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d999cfac-10cf-48ea-bd09-ca9ef65a68ae service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:29.001975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d999cfac-10cf-48ea-bd09-ca9ef65a68ae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:29.063269 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d999cfac-10cf-48ea-bd09-ca9ef65a68ae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:29.064637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d999cfac-10cf-48ea-bd09-ca9ef65a68ae service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:29.074455 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1346/2711] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:35:28 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:35:31.867692 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f77b78-dc5e-417e-89d0-ebce450a0a96 service cinder] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:31.891102 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:31.947035 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f77b78-dc5e-417e-89d0-ebce450a0a96 service cinder] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:31.952340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb1bc3a-e2fd-4689-8f0b-31e222829c99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:31.952978 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb1bc3a-e2fd-4689-8f0b-31e222829c99 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:31.953363 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb1bc3a-e2fd-4689-8f0b-31e222829c99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:31.953688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb1bc3a-e2fd-4689-8f0b-31e222829c99 admin admin] PATH_INFO: `/v3/users/89f3ae28f5d145979d2d2403e50c8807` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:31.955239 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb1bc3a-e2fd-4689-8f0b-31e222829c99 admin admin] RBAC: Authorizing `identity:delete_user(user_id=89f3ae28f5d145979d2d2403e50c8807)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:31.957882 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb1bc3a-e2fd-4689-8f0b-31e222829c99 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:32.006825 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-0fb1bc3a-e2fd-4689-8f0b-31e222829c99 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89f3ae28f5d145979d2d2403e50c8807'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:32.014615 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-0fb1bc3a-e2fd-4689-8f0b-31e222829c99 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89f3ae28f5d145979d2d2403e50c8807'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:32.016505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-0fb1bc3a-e2fd-4689-8f0b-31e222829c99 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89f3ae28f5d145979d2d2403e50c8807'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:32.030123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-0fb1bc3a-e2fd-4689-8f0b-31e222829c99 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89f3ae28f5d145979d2d2403e50c8807'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:32.030793 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-0fb1bc3a-e2fd-4689-8f0b-31e222829c99 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89f3ae28f5d145979d2d2403e50c8807'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:32.043698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1366/2712] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:35:31 2022] DELETE /identity/v3/users/89f3ae28f5d145979d2d2403e50c8807 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:35:32.048514 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da837fc-f262-4ad2-847a-e3ea578bf124 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:32.049019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da837fc-f262-4ad2-847a-e3ea578bf124 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:32.049233 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da837fc-f262-4ad2-847a-e3ea578bf124 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:32.068845 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4da837fc-f262-4ad2-847a-e3ea578bf124 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:32.070098 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4da837fc-f262-4ad2-847a-e3ea578bf124 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:32.132414 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1347/2713] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:32.142704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb1bc3a-e2fd-4689-8f0b-31e222829c99 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:32.144154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-0fb1bc3a-e2fd-4689-8f0b-31e222829c99 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:32.205739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb1bc3a-e2fd-4689-8f0b-31e222829c99 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:32.209448 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e2a303-5e16-4b0a-a60e-45c32565ff17 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:32.210007 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e2a303-5e16-4b0a-a60e-45c32565ff17 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:32.210288 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e2a303-5e16-4b0a-a60e-45c32565ff17 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:32.210565 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e2a303-5e16-4b0a-a60e-45c32565ff17 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:32.219502 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e2a303-5e16-4b0a-a60e-45c32565ff17 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:32.220921 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e2a303-5e16-4b0a-a60e-45c32565ff17 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:32.232471 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1367/2714] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:35:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:35:32.499621 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da837fc-f262-4ad2-847a-e3ea578bf124 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:32.501091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4da837fc-f262-4ad2-847a-e3ea578bf124 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:32.513196 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da837fc-f262-4ad2-847a-e3ea578bf124 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:32.517867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14221878-2233-469c-9f9e-4acd2ab36a62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:32.518550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14221878-2233-469c-9f9e-4acd2ab36a62 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:32.518827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14221878-2233-469c-9f9e-4acd2ab36a62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:32.519103 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14221878-2233-469c-9f9e-4acd2ab36a62 admin admin] PATH_INFO: `/v3/projects/043212768c2d44a283ab1eeabc295ba7` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:32.520236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14221878-2233-469c-9f9e-4acd2ab36a62 admin admin] RBAC: Authorizing `identity:delete_project(project_id=043212768c2d44a283ab1eeabc295ba7)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:32.522202 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14221878-2233-469c-9f9e-4acd2ab36a62 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:32.569315 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-14221878-2233-469c-9f9e-4acd2ab36a62 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '043212768c2d44a283ab1eeabc295ba7', 'request_id': 'req-14221878-2233-469c-9f9e-4acd2ab36a62'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:32.574912 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-14221878-2233-469c-9f9e-4acd2ab36a62 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '043212768c2d44a283ab1eeabc295ba7', 'request_id': 'req-14221878-2233-469c-9f9e-4acd2ab36a62'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:32.582978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-14221878-2233-469c-9f9e-4acd2ab36a62 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '043212768c2d44a283ab1eeabc295ba7', 'request_id': 'req-14221878-2233-469c-9f9e-4acd2ab36a62'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:32.595713 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-14221878-2233-469c-9f9e-4acd2ab36a62 admin admin] The token cache is being invalidate because project 043212768c2d44a283ab1eeabc295ba7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:35:32.596010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-14221878-2233-469c-9f9e-4acd2ab36a62 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:32.596756 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-14221878-2233-469c-9f9e-4acd2ab36a62 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:32.598300 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1348/2715] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:35:32 2022] DELETE /identity/v3/projects/043212768c2d44a283ab1eeabc295ba7 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:32.602504 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e2a303-5e16-4b0a-a60e-45c32565ff17 service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:32.613144 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e2a303-5e16-4b0a-a60e-45c32565ff17 service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:32.654411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:32.674001 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e2faac-5825-43fe-9874-69a0f9f378a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:32.674427 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e2faac-5825-43fe-9874-69a0f9f378a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:32.674706 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e2faac-5825-43fe-9874-69a0f9f378a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:32.674977 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e2faac-5825-43fe-9874-69a0f9f378a7 admin admin] PATH_INFO: `/v3/users/faf34a5ebdcc4d959264c4649ffae5de` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:32.676096 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e2faac-5825-43fe-9874-69a0f9f378a7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=faf34a5ebdcc4d959264c4649ffae5de)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:32.678067 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e2faac-5825-43fe-9874-69a0f9f378a7 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:32.729695 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-15e2faac-5825-43fe-9874-69a0f9f378a7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'faf34a5ebdcc4d959264c4649ffae5de'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:32.738052 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-15e2faac-5825-43fe-9874-69a0f9f378a7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'faf34a5ebdcc4d959264c4649ffae5de'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:32.738707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-15e2faac-5825-43fe-9874-69a0f9f378a7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'faf34a5ebdcc4d959264c4649ffae5de'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:32.752100 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-15e2faac-5825-43fe-9874-69a0f9f378a7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'faf34a5ebdcc4d959264c4649ffae5de'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:32.752775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-15e2faac-5825-43fe-9874-69a0f9f378a7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'faf34a5ebdcc4d959264c4649ffae5de'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:32.764782 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1368/2716] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:35:32 2022] DELETE /identity/v3/users/faf34a5ebdcc4d959264c4649ffae5de => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:33.039336 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14221878-2233-469c-9f9e-4acd2ab36a62 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:33.040780 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-14221878-2233-469c-9f9e-4acd2ab36a62 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:33.056422 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14221878-2233-469c-9f9e-4acd2ab36a62 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:33.075446 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:33.118218 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e0dcd0-e34f-42b0-a00f-27420b4a7a47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:33.118871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e0dcd0-e34f-42b0-a00f-27420b4a7a47 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:33.119194 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e0dcd0-e34f-42b0-a00f-27420b4a7a47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:33.119448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e0dcd0-e34f-42b0-a00f-27420b4a7a47 admin admin] PATH_INFO: `/v3/projects/6f1174a665b248e09e29358cecdd048f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:33.120628 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e0dcd0-e34f-42b0-a00f-27420b4a7a47 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6f1174a665b248e09e29358cecdd048f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:33.122638 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e0dcd0-e34f-42b0-a00f-27420b4a7a47 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:33.300414 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-60e0dcd0-e34f-42b0-a00f-27420b4a7a47 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6f1174a665b248e09e29358cecdd048f', 'request_id': 'req-60e0dcd0-e34f-42b0-a00f-27420b4a7a47'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:33.317910 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-60e0dcd0-e34f-42b0-a00f-27420b4a7a47 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6f1174a665b248e09e29358cecdd048f', 'request_id': 'req-60e0dcd0-e34f-42b0-a00f-27420b4a7a47'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:33.326871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-60e0dcd0-e34f-42b0-a00f-27420b4a7a47 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6f1174a665b248e09e29358cecdd048f', 'request_id': 'req-60e0dcd0-e34f-42b0-a00f-27420b4a7a47'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:33.340664 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-60e0dcd0-e34f-42b0-a00f-27420b4a7a47 admin admin] The token cache is being invalidate because project 6f1174a665b248e09e29358cecdd048f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:35:33.340954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-60e0dcd0-e34f-42b0-a00f-27420b4a7a47 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:33.342100 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-60e0dcd0-e34f-42b0-a00f-27420b4a7a47 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:33.343258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1349/2717] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:35:33 2022] DELETE /identity/v3/projects/6f1174a665b248e09e29358cecdd048f => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:33.348890 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8293da4a-3453-4687-9f39-8d02f3f14a09 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:33.349419 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8293da4a-3453-4687-9f39-8d02f3f14a09 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:33.349505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8293da4a-3453-4687-9f39-8d02f3f14a09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:33.491828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-8293da4a-3453-4687-9f39-8d02f3f14a09 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:33.493071 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8293da4a-3453-4687-9f39-8d02f3f14a09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:33.552620 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1369/2718] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:33.560460 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a070232-a44d-4ea5-8e39-6ad93ef566d3 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:33.560774 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a070232-a44d-4ea5-8e39-6ad93ef566d3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:33.561064 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a070232-a44d-4ea5-8e39-6ad93ef566d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:33.581086 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4a070232-a44d-4ea5-8e39-6ad93ef566d3 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:33.582318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4a070232-a44d-4ea5-8e39-6ad93ef566d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:33.590746 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1350/2719] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:33.595038 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8293da4a-3453-4687-9f39-8d02f3f14a09 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:33.605121 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8293da4a-3453-4687-9f39-8d02f3f14a09 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:33.610131 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edbde0d-2590-4d2f-a2dc-cf3f0f57519a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:33.610786 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edbde0d-2590-4d2f-a2dc-cf3f0f57519a admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:33.611064 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edbde0d-2590-4d2f-a2dc-cf3f0f57519a admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:33.611484 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edbde0d-2590-4d2f-a2dc-cf3f0f57519a admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:33.612129 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6edbde0d-2590-4d2f-a2dc-cf3f0f57519a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:35:33.612429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6edbde0d-2590-4d2f-a2dc-cf3f0f57519a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:33.614044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6edbde0d-2590-4d2f-a2dc-cf3f0f57519a admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:33.626659 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1370/2720] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:35:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:35:33.631364 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a070232-a44d-4ea5-8e39-6ad93ef566d3 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:33.641301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a070232-a44d-4ea5-8e39-6ad93ef566d3 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:33.645892 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7fb6a8-6689-4b3e-9d96-73624c179640 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:33.646488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7fb6a8-6689-4b3e-9d96-73624c179640 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:33.646766 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7fb6a8-6689-4b3e-9d96-73624c179640 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:33.647034 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7fb6a8-6689-4b3e-9d96-73624c179640 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:33.647677 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7fb6a8-6689-4b3e-9d96-73624c179640 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:33.649609 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7fb6a8-6689-4b3e-9d96-73624c179640 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:33.675789 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1351/2721] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:35:33 2022] POST /identity/v3/projects => generated 387 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:35:33.680008 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edbde0d-2590-4d2f-a2dc-cf3f0f57519a admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:33.690251 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edbde0d-2590-4d2f-a2dc-cf3f0f57519a admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:33.733575 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:33.747445 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460f471a-c3be-4129-9417-071f219ea6af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:33.748043 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460f471a-c3be-4129-9417-071f219ea6af admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:33.748312 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460f471a-c3be-4129-9417-071f219ea6af admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:33.748578 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460f471a-c3be-4129-9417-071f219ea6af admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:33.749247 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460f471a-c3be-4129-9417-071f219ea6af admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:33.751118 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460f471a-c3be-4129-9417-071f219ea6af admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:33.775966 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1371/2722] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:35:33 2022] POST /identity/v3/users => generated 294 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:35:33.777572 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7fb6a8-6689-4b3e-9d96-73624c179640 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:33.788132 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7fb6a8-6689-4b3e-9d96-73624c179640 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:33.792817 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e225c4b-6d38-4f41-aa73-9efafcc56445 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:33.793478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e225c4b-6d38-4f41-aa73-9efafcc56445 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:33.793760 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e225c4b-6d38-4f41-aa73-9efafcc56445 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:33.794059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e225c4b-6d38-4f41-aa73-9efafcc56445 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:33.794780 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e225c4b-6d38-4f41-aa73-9efafcc56445 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:33.796515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e225c4b-6d38-4f41-aa73-9efafcc56445 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:33.807814 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1352/2723] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:33 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:33.811923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460f471a-c3be-4129-9417-071f219ea6af admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:33.822238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460f471a-c3be-4129-9417-071f219ea6af admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:33.827522 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b494ed-12d2-4c76-a0f8-82945f1d63c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:33.828125 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b494ed-12d2-4c76-a0f8-82945f1d63c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:33.828405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b494ed-12d2-4c76-a0f8-82945f1d63c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:33.828677 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b494ed-12d2-4c76-a0f8-82945f1d63c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:33.829366 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b494ed-12d2-4c76-a0f8-82945f1d63c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:33.830967 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b494ed-12d2-4c76-a0f8-82945f1d63c5 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:33.841228 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1372/2724] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:33 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:35:33.846800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e225c4b-6d38-4f41-aa73-9efafcc56445 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:33.857404 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e225c4b-6d38-4f41-aa73-9efafcc56445 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:33.862444 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb7cdca-0ff0-4229-8f38-90cbe6d339fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:33.863106 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb7cdca-0ff0-4229-8f38-90cbe6d339fb admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:33.863419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb7cdca-0ff0-4229-8f38-90cbe6d339fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:33.863693 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb7cdca-0ff0-4229-8f38-90cbe6d339fb admin admin] PATH_INFO: `/v3/projects/bb0d09fe4ff3418ea14d8d79cab3e1cc/users/e52b77119fb74753bb04009c8f60f88f/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:33.879856 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb7cdca-0ff0-4229-8f38-90cbe6d339fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb0d09fe4ff3418ea14d8d79cab3e1cc, user_id=e52b77119fb74753bb04009c8f60f88f, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:33.882500 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb7cdca-0ff0-4229-8f38-90cbe6d339fb admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:33.901027 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1353/2725] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:35:33 2022] PUT /identity/v3/projects/bb0d09fe4ff3418ea14d8d79cab3e1cc/users/e52b77119fb74753bb04009c8f60f88f/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:33.911771 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1553a746-45ac-478f-8cd1-473f71efaff0 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:33.912371 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1553a746-45ac-478f-8cd1-473f71efaff0 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:33.912639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1553a746-45ac-478f-8cd1-473f71efaff0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:33.942365 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-1553a746-45ac-478f-8cd1-473f71efaff0 None None] MFA Rules not processed for user `e52b77119fb74753bb04009c8f60f88f`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:33.943559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-1553a746-45ac-478f-8cd1-473f71efaff0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:34.006975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1373/2726] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:33 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:34.011679 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb7cdca-0ff0-4229-8f38-90cbe6d339fb admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:34.059650 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:34.074742 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb7cdca-0ff0-4229-8f38-90cbe6d339fb admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:34.079473 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34dde04-6f52-4f97-9761-f598647e7766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:34.080111 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34dde04-6f52-4f97-9761-f598647e7766 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:34.080395 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34dde04-6f52-4f97-9761-f598647e7766 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:34.080672 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34dde04-6f52-4f97-9761-f598647e7766 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:34.081372 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34dde04-6f52-4f97-9761-f598647e7766 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:34.083323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34dde04-6f52-4f97-9761-f598647e7766 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:34.111001 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1354/2727] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:35:34 2022] POST /identity/v3/projects => generated 387 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:35:34.114962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1553a746-45ac-478f-8cd1-473f71efaff0 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:34.126476 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1553a746-45ac-478f-8cd1-473f71efaff0 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:34.183860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23701c1-de96-4bfe-b325-19726297bf64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:34.184415 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23701c1-de96-4bfe-b325-19726297bf64 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:34.184691 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23701c1-de96-4bfe-b325-19726297bf64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:34.184962 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23701c1-de96-4bfe-b325-19726297bf64 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:34.185897 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23701c1-de96-4bfe-b325-19726297bf64 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:34.187718 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23701c1-de96-4bfe-b325-19726297bf64 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:34.209715 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1374/2728] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:35:34 2022] POST /identity/v3/users => generated 300 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:35:34.214962 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34dde04-6f52-4f97-9761-f598647e7766 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:34.225090 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34dde04-6f52-4f97-9761-f598647e7766 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:34.230142 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9cb9c5-4f75-4be9-83f4-61b1d5bbbc4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:34.230748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9cb9c5-4f75-4be9-83f4-61b1d5bbbc4e admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:34.231030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9cb9c5-4f75-4be9-83f4-61b1d5bbbc4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:34.231302 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9cb9c5-4f75-4be9-83f4-61b1d5bbbc4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:34.231945 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9cb9c5-4f75-4be9-83f4-61b1d5bbbc4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:34.233573 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9cb9c5-4f75-4be9-83f4-61b1d5bbbc4e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:34.244048 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1355/2729] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:34 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:34.249361 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23701c1-de96-4bfe-b325-19726297bf64 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:34.259428 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23701c1-de96-4bfe-b325-19726297bf64 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:34.264054 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5692daa2-4a7d-440d-a929-441aa1dc6460 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:34.264705 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5692daa2-4a7d-440d-a929-441aa1dc6460 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:34.265003 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5692daa2-4a7d-440d-a929-441aa1dc6460 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:34.265289 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5692daa2-4a7d-440d-a929-441aa1dc6460 admin admin] PATH_INFO: `/v3/projects/c6b76e8d07f544a6bac2d2dc02bd279a/users/212e63ffd0424d458b24f773572fc914/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:34.280046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5692daa2-4a7d-440d-a929-441aa1dc6460 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6b76e8d07f544a6bac2d2dc02bd279a, user_id=212e63ffd0424d458b24f773572fc914, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:34.282725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5692daa2-4a7d-440d-a929-441aa1dc6460 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:34.295814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1375/2730] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:35:34 2022] PUT /identity/v3/projects/c6b76e8d07f544a6bac2d2dc02bd279a/users/212e63ffd0424d458b24f773572fc914/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:34.299904 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9cb9c5-4f75-4be9-83f4-61b1d5bbbc4e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:34.310516 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9cb9c5-4f75-4be9-83f4-61b1d5bbbc4e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:34.368816 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1467f51c-8069-4bc4-9746-f2c18dc685c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:34.369558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1467f51c-8069-4bc4-9746-f2c18dc685c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:34.369863 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1467f51c-8069-4bc4-9746-f2c18dc685c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:34.370157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1467f51c-8069-4bc4-9746-f2c18dc685c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:34.370808 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1467f51c-8069-4bc4-9746-f2c18dc685c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:34.372491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1467f51c-8069-4bc4-9746-f2c18dc685c8 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:34.384304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1356/2731] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:34 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:34.388894 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5692daa2-4a7d-440d-a929-441aa1dc6460 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:34.400181 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5692daa2-4a7d-440d-a929-441aa1dc6460 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:34.405703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cece97d-010d-4efc-85e3-56af81860cc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:34.406381 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cece97d-010d-4efc-85e3-56af81860cc3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:34.406657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cece97d-010d-4efc-85e3-56af81860cc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:34.406932 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cece97d-010d-4efc-85e3-56af81860cc3 admin admin] PATH_INFO: `/v3/projects/c6b76e8d07f544a6bac2d2dc02bd279a/users/212e63ffd0424d458b24f773572fc914/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:34.409165 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cece97d-010d-4efc-85e3-56af81860cc3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6b76e8d07f544a6bac2d2dc02bd279a, user_id=212e63ffd0424d458b24f773572fc914, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:34.411707 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cece97d-010d-4efc-85e3-56af81860cc3 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:34.423904 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1376/2732] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:35:34 2022] PUT /identity/v3/projects/c6b76e8d07f544a6bac2d2dc02bd279a/users/212e63ffd0424d458b24f773572fc914/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:35:34.434376 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f0d9b2-cf23-4100-9c94-1dad2e0eb708 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:34.434658 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f0d9b2-cf23-4100-9c94-1dad2e0eb708 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:34.434926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f0d9b2-cf23-4100-9c94-1dad2e0eb708 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:34.467047 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-48f0d9b2-cf23-4100-9c94-1dad2e0eb708 None None] MFA Rules not processed for user `212e63ffd0424d458b24f773572fc914`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:34.468344 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-48f0d9b2-cf23-4100-9c94-1dad2e0eb708 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:34.540929 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1357/2733] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:34 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:34.553252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cece97d-010d-4efc-85e3-56af81860cc3 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:34.554937 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-8cece97d-010d-4efc-85e3-56af81860cc3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:34.566305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:34.614620 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cece97d-010d-4efc-85e3-56af81860cc3 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:34.618320 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f53be47-3549-4a6d-b300-ec374464d51f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:34.618925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f53be47-3549-4a6d-b300-ec374464d51f service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:34.619217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f53be47-3549-4a6d-b300-ec374464d51f service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:34.619489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f53be47-3549-4a6d-b300-ec374464d51f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:34.628461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f53be47-3549-4a6d-b300-ec374464d51f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:34.629998 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f53be47-3549-4a6d-b300-ec374464d51f service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:34.639333 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1377/2734] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:35:34 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:35:34.706179 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f0d9b2-cf23-4100-9c94-1dad2e0eb708 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:34.751698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:34.769648 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f0d9b2-cf23-4100-9c94-1dad2e0eb708 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:34.774363 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f652cd-d3fe-4593-977d-624891df2e17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:34.774997 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f652cd-d3fe-4593-977d-624891df2e17 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:34.775276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f652cd-d3fe-4593-977d-624891df2e17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:34.775546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f652cd-d3fe-4593-977d-624891df2e17 admin admin] PATH_INFO: `/v3/users/e52b77119fb74753bb04009c8f60f88f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:34.777323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f652cd-d3fe-4593-977d-624891df2e17 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e52b77119fb74753bb04009c8f60f88f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:34.779248 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f652cd-d3fe-4593-977d-624891df2e17 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:34.827023 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-23f652cd-d3fe-4593-977d-624891df2e17 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e52b77119fb74753bb04009c8f60f88f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:34.839280 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-23f652cd-d3fe-4593-977d-624891df2e17 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e52b77119fb74753bb04009c8f60f88f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:34.846514 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-23f652cd-d3fe-4593-977d-624891df2e17 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e52b77119fb74753bb04009c8f60f88f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:34.853245 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-23f652cd-d3fe-4593-977d-624891df2e17 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e52b77119fb74753bb04009c8f60f88f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:34.853865 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-23f652cd-d3fe-4593-977d-624891df2e17 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e52b77119fb74753bb04009c8f60f88f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:34.860014 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1358/2735] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:35:34 2022] DELETE /identity/v3/users/e52b77119fb74753bb04009c8f60f88f => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:34.864519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af93df22-df5f-4405-b6d0-e33df18897d2 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:34.864805 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af93df22-df5f-4405-b6d0-e33df18897d2 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:34.865099 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af93df22-df5f-4405-b6d0-e33df18897d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:34.883454 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-af93df22-df5f-4405-b6d0-e33df18897d2 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:34.884645 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-af93df22-df5f-4405-b6d0-e33df18897d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:34.943944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1378/2736] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:34.954193 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f652cd-d3fe-4593-977d-624891df2e17 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:34.955708 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-23f652cd-d3fe-4593-977d-624891df2e17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:35.016745 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f652cd-d3fe-4593-977d-624891df2e17 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:35.020623 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f21c14-9dea-476f-8e7d-55f1fd0e1af2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:35.021276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f21c14-9dea-476f-8e7d-55f1fd0e1af2 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:35.021578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f21c14-9dea-476f-8e7d-55f1fd0e1af2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:35.021850 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f21c14-9dea-476f-8e7d-55f1fd0e1af2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:35.031942 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f21c14-9dea-476f-8e7d-55f1fd0e1af2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:35.033429 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f21c14-9dea-476f-8e7d-55f1fd0e1af2 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:35.045501 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1359/2737] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:35:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:35:35.384403 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af93df22-df5f-4405-b6d0-e33df18897d2 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:35.387531 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-af93df22-df5f-4405-b6d0-e33df18897d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:35.406273 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af93df22-df5f-4405-b6d0-e33df18897d2 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:35.410787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655f4b8a-0bb5-4a6c-a010-b6926ee75fe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:35.411414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655f4b8a-0bb5-4a6c-a010-b6926ee75fe0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:35.411691 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655f4b8a-0bb5-4a6c-a010-b6926ee75fe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:35.411954 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655f4b8a-0bb5-4a6c-a010-b6926ee75fe0 admin admin] PATH_INFO: `/v3/projects/bb0d09fe4ff3418ea14d8d79cab3e1cc` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:35.413063 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655f4b8a-0bb5-4a6c-a010-b6926ee75fe0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb0d09fe4ff3418ea14d8d79cab3e1cc)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:35.416161 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655f4b8a-0bb5-4a6c-a010-b6926ee75fe0 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:35.477621 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-655f4b8a-0bb5-4a6c-a010-b6926ee75fe0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb0d09fe4ff3418ea14d8d79cab3e1cc', 'request_id': 'req-655f4b8a-0bb5-4a6c-a010-b6926ee75fe0'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:35.484267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-655f4b8a-0bb5-4a6c-a010-b6926ee75fe0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb0d09fe4ff3418ea14d8d79cab3e1cc', 'request_id': 'req-655f4b8a-0bb5-4a6c-a010-b6926ee75fe0'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:35.492359 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-655f4b8a-0bb5-4a6c-a010-b6926ee75fe0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb0d09fe4ff3418ea14d8d79cab3e1cc', 'request_id': 'req-655f4b8a-0bb5-4a6c-a010-b6926ee75fe0'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:35.504002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-655f4b8a-0bb5-4a6c-a010-b6926ee75fe0 admin admin] The token cache is being invalidate because project bb0d09fe4ff3418ea14d8d79cab3e1cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:35:35.504287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-655f4b8a-0bb5-4a6c-a010-b6926ee75fe0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:35.504904 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-655f4b8a-0bb5-4a6c-a010-b6926ee75fe0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:35.506410 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1379/2738] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:35:35 2022] DELETE /identity/v3/projects/bb0d09fe4ff3418ea14d8d79cab3e1cc => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:35.510943 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f21c14-9dea-476f-8e7d-55f1fd0e1af2 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:35.520347 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f21c14-9dea-476f-8e7d-55f1fd0e1af2 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:35.576913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692bb8c9-7ccf-4da8-b7f4-f5911648ab6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:35.577579 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692bb8c9-7ccf-4da8-b7f4-f5911648ab6e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:35.577862 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692bb8c9-7ccf-4da8-b7f4-f5911648ab6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:35.578138 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692bb8c9-7ccf-4da8-b7f4-f5911648ab6e admin admin] PATH_INFO: `/v3/users/212e63ffd0424d458b24f773572fc914` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:35.579264 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692bb8c9-7ccf-4da8-b7f4-f5911648ab6e admin admin] RBAC: Authorizing `identity:delete_user(user_id=212e63ffd0424d458b24f773572fc914)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:35.581208 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692bb8c9-7ccf-4da8-b7f4-f5911648ab6e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:35.586828 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:35.631726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-692bb8c9-7ccf-4da8-b7f4-f5911648ab6e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '212e63ffd0424d458b24f773572fc914'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:35.643964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-692bb8c9-7ccf-4da8-b7f4-f5911648ab6e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '212e63ffd0424d458b24f773572fc914'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:35.651231 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-692bb8c9-7ccf-4da8-b7f4-f5911648ab6e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '212e63ffd0424d458b24f773572fc914'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:35.658489 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-692bb8c9-7ccf-4da8-b7f4-f5911648ab6e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '212e63ffd0424d458b24f773572fc914'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:35.660051 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-692bb8c9-7ccf-4da8-b7f4-f5911648ab6e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '212e63ffd0424d458b24f773572fc914'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:35.667087 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1360/2739] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:35:35 2022] DELETE /identity/v3/users/212e63ffd0424d458b24f773572fc914 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:36.003048 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655f4b8a-0bb5-4a6c-a010-b6926ee75fe0 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:36.004481 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-655f4b8a-0bb5-4a6c-a010-b6926ee75fe0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:36.019450 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655f4b8a-0bb5-4a6c-a010-b6926ee75fe0 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:36.053922 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:36.079276 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85c7f87-054c-4178-8b85-f5b82f3110d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:36.079918 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85c7f87-054c-4178-8b85-f5b82f3110d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:36.080194 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85c7f87-054c-4178-8b85-f5b82f3110d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:36.080462 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85c7f87-054c-4178-8b85-f5b82f3110d2 admin admin] PATH_INFO: `/v3/projects/c6b76e8d07f544a6bac2d2dc02bd279a` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:36.081584 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85c7f87-054c-4178-8b85-f5b82f3110d2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c6b76e8d07f544a6bac2d2dc02bd279a)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:36.083481 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85c7f87-054c-4178-8b85-f5b82f3110d2 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:36.130746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a85c7f87-054c-4178-8b85-f5b82f3110d2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c6b76e8d07f544a6bac2d2dc02bd279a', 'request_id': 'req-a85c7f87-054c-4178-8b85-f5b82f3110d2'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:36.136146 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a85c7f87-054c-4178-8b85-f5b82f3110d2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c6b76e8d07f544a6bac2d2dc02bd279a', 'request_id': 'req-a85c7f87-054c-4178-8b85-f5b82f3110d2'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:36.143308 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a85c7f87-054c-4178-8b85-f5b82f3110d2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c6b76e8d07f544a6bac2d2dc02bd279a', 'request_id': 'req-a85c7f87-054c-4178-8b85-f5b82f3110d2'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:36.156042 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a85c7f87-054c-4178-8b85-f5b82f3110d2 admin admin] The token cache is being invalidate because project c6b76e8d07f544a6bac2d2dc02bd279a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:35:36.156317 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a85c7f87-054c-4178-8b85-f5b82f3110d2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:36.156988 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a85c7f87-054c-4178-8b85-f5b82f3110d2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:36.158507 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1380/2740] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:35:36 2022] DELETE /identity/v3/projects/c6b76e8d07f544a6bac2d2dc02bd279a => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:36.163649 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366719f9-58e3-45a3-b76a-7a3134b01aa2 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:36.163995 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366719f9-58e3-45a3-b76a-7a3134b01aa2 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:36.164394 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366719f9-58e3-45a3-b76a-7a3134b01aa2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:36.185682 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-366719f9-58e3-45a3-b76a-7a3134b01aa2 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:36.186896 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-366719f9-58e3-45a3-b76a-7a3134b01aa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:36.253150 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1361/2741] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:36.262479 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108e11a6-5106-4be9-bda6-5ba8c04bdc87 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:36.262765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108e11a6-5106-4be9-bda6-5ba8c04bdc87 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:36.263055 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108e11a6-5106-4be9-bda6-5ba8c04bdc87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:36.283533 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-108e11a6-5106-4be9-bda6-5ba8c04bdc87 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:36.284666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-108e11a6-5106-4be9-bda6-5ba8c04bdc87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:36.293271 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1381/2742] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:36.296806 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366719f9-58e3-45a3-b76a-7a3134b01aa2 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:36.310160 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366719f9-58e3-45a3-b76a-7a3134b01aa2 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:36.314920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f31f20-9dca-4948-9f83-2abdc2f15fd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:36.315522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f31f20-9dca-4948-9f83-2abdc2f15fd2 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:36.315800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f31f20-9dca-4948-9f83-2abdc2f15fd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:36.316073 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f31f20-9dca-4948-9f83-2abdc2f15fd2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:36.316723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f31f20-9dca-4948-9f83-2abdc2f15fd2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:35:36.317025 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f31f20-9dca-4948-9f83-2abdc2f15fd2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:36.318685 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f31f20-9dca-4948-9f83-2abdc2f15fd2 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:36.331491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1362/2743] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:35:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:35:36.336505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108e11a6-5106-4be9-bda6-5ba8c04bdc87 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:36.346425 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108e11a6-5106-4be9-bda6-5ba8c04bdc87 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:36.351167 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bddd4d2b-8df1-4561-ab41-8ec63f1a8075 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:36.351757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddd4d2b-8df1-4561-ab41-8ec63f1a8075 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:36.352036 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddd4d2b-8df1-4561-ab41-8ec63f1a8075 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:36.352310 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddd4d2b-8df1-4561-ab41-8ec63f1a8075 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:36.352934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bddd4d2b-8df1-4561-ab41-8ec63f1a8075 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:36.355002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bddd4d2b-8df1-4561-ab41-8ec63f1a8075 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:36.381541 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1382/2744] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:35:36 2022] POST /identity/v3/projects => generated 379 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:35:36.386313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f31f20-9dca-4948-9f83-2abdc2f15fd2 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:36.396078 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f31f20-9dca-4948-9f83-2abdc2f15fd2 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:36.452030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a753f97-d0f5-42f7-8dc7-1cbc4ec026cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:36.452644 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a753f97-d0f5-42f7-8dc7-1cbc4ec026cf admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:36.452926 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a753f97-d0f5-42f7-8dc7-1cbc4ec026cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:36.453244 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a753f97-d0f5-42f7-8dc7-1cbc4ec026cf admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:36.453877 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a753f97-d0f5-42f7-8dc7-1cbc4ec026cf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:36.455854 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a753f97-d0f5-42f7-8dc7-1cbc4ec026cf admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:36.476986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1363/2745] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:35:36 2022] POST /identity/v3/users => generated 290 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:35:36.481318 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bddd4d2b-8df1-4561-ab41-8ec63f1a8075 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:36.491860 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bddd4d2b-8df1-4561-ab41-8ec63f1a8075 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:36.496857 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a72812-0d1d-4d7a-816c-faa0356916c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:36.497559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a72812-0d1d-4d7a-816c-faa0356916c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:36.497840 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a72812-0d1d-4d7a-816c-faa0356916c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:36.498374 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a72812-0d1d-4d7a-816c-faa0356916c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:36.499028 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a72812-0d1d-4d7a-816c-faa0356916c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:36.500671 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a72812-0d1d-4d7a-816c-faa0356916c3 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:36.510993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1383/2746] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:36.514958 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a753f97-d0f5-42f7-8dc7-1cbc4ec026cf admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:36.520208 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:36.527176 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a753f97-d0f5-42f7-8dc7-1cbc4ec026cf admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:36.532671 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f12181-04db-4494-a986-f611dc6d53ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:36.533316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f12181-04db-4494-a986-f611dc6d53ff admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:36.533646 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f12181-04db-4494-a986-f611dc6d53ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:36.533876 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f12181-04db-4494-a986-f611dc6d53ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:36.534524 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f12181-04db-4494-a986-f611dc6d53ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:36.536117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f12181-04db-4494-a986-f611dc6d53ff admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:36.546932 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1364/2747] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:36.551119 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a72812-0d1d-4d7a-816c-faa0356916c3 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:36.561102 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a72812-0d1d-4d7a-816c-faa0356916c3 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:36.566238 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e723a618-188a-48de-a5a2-961911b30dca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:36.566878 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e723a618-188a-48de-a5a2-961911b30dca admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:36.567158 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e723a618-188a-48de-a5a2-961911b30dca admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:36.567423 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e723a618-188a-48de-a5a2-961911b30dca admin admin] PATH_INFO: `/v3/projects/3918d59d318945a9b3ec45b7704cad66/users/dbd17e217fb944ee81a6c39920a439e0/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:36.582222 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e723a618-188a-48de-a5a2-961911b30dca admin admin] RBAC: Authorizing `identity:create_grant(project_id=3918d59d318945a9b3ec45b7704cad66, user_id=dbd17e217fb944ee81a6c39920a439e0, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:36.584828 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e723a618-188a-48de-a5a2-961911b30dca admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:36.598052 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1384/2748] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:35:36 2022] PUT /identity/v3/projects/3918d59d318945a9b3ec45b7704cad66/users/dbd17e217fb944ee81a6c39920a439e0/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:36.608426 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84e6e76-5d99-42df-8092-6bfbfb93561a None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:36.608971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84e6e76-5d99-42df-8092-6bfbfb93561a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:36.609279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84e6e76-5d99-42df-8092-6bfbfb93561a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:36.638013 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b84e6e76-5d99-42df-8092-6bfbfb93561a None None] MFA Rules not processed for user `dbd17e217fb944ee81a6c39920a439e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:36.639204 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b84e6e76-5d99-42df-8092-6bfbfb93561a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:36.701056 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1365/2749] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:36 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:36.705730 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e723a618-188a-48de-a5a2-961911b30dca admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:36.766023 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e723a618-188a-48de-a5a2-961911b30dca admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:36.770755 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a6eaa1-1de8-4c7d-be0b-677b27dc3826 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:36.771360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a6eaa1-1de8-4c7d-be0b-677b27dc3826 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:36.771639 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a6eaa1-1de8-4c7d-be0b-677b27dc3826 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:36.771909 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a6eaa1-1de8-4c7d-be0b-677b27dc3826 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:36.772544 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a6eaa1-1de8-4c7d-be0b-677b27dc3826 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:36.774461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a6eaa1-1de8-4c7d-be0b-677b27dc3826 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:36.780454 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:36.809419 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1385/2750] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:35:36 2022] POST /identity/v3/projects => generated 379 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:35:36.811254 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84e6e76-5d99-42df-8092-6bfbfb93561a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:36.822083 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84e6e76-5d99-42df-8092-6bfbfb93561a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:36.883206 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f1be54-9915-4099-8d2e-fecdd97a01c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:36.883737 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f1be54-9915-4099-8d2e-fecdd97a01c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:36.884039 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f1be54-9915-4099-8d2e-fecdd97a01c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:36.884323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f1be54-9915-4099-8d2e-fecdd97a01c6 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:36.884946 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f1be54-9915-4099-8d2e-fecdd97a01c6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:36.887492 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f1be54-9915-4099-8d2e-fecdd97a01c6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:36.909654 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1366/2751] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:35:36 2022] POST /identity/v3/users => generated 296 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:35:36.913360 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a6eaa1-1de8-4c7d-be0b-677b27dc3826 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:36.923747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a6eaa1-1de8-4c7d-be0b-677b27dc3826 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:36.928659 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7563f485-5da6-4011-af97-d1df2970fc38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:36.929373 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7563f485-5da6-4011-af97-d1df2970fc38 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:36.929654 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7563f485-5da6-4011-af97-d1df2970fc38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:36.930006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7563f485-5da6-4011-af97-d1df2970fc38 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:36.930575 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7563f485-5da6-4011-af97-d1df2970fc38 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:36.932725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7563f485-5da6-4011-af97-d1df2970fc38 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:36.943675 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1386/2752] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:36 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:36.947568 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f1be54-9915-4099-8d2e-fecdd97a01c6 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:36.958838 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f1be54-9915-4099-8d2e-fecdd97a01c6 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:36.963458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cdf680-3f7a-4389-afed-31356a0d0f00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:36.964103 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cdf680-3f7a-4389-afed-31356a0d0f00 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:36.964379 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cdf680-3f7a-4389-afed-31356a0d0f00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:36.964652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cdf680-3f7a-4389-afed-31356a0d0f00 admin admin] PATH_INFO: `/v3/projects/daebf2d0549c4c6592da9d8a0b878146/users/0954bbf7ae0642b894b21e09a9f2874e/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:36.980228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cdf680-3f7a-4389-afed-31356a0d0f00 admin admin] RBAC: Authorizing `identity:create_grant(project_id=daebf2d0549c4c6592da9d8a0b878146, user_id=0954bbf7ae0642b894b21e09a9f2874e, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:36.982906 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cdf680-3f7a-4389-afed-31356a0d0f00 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:36.996377 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1367/2753] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:35:36 2022] PUT /identity/v3/projects/daebf2d0549c4c6592da9d8a0b878146/users/0954bbf7ae0642b894b21e09a9f2874e/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:37.000332 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7563f485-5da6-4011-af97-d1df2970fc38 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:37.010747 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7563f485-5da6-4011-af97-d1df2970fc38 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:37.066580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471b8d46-be46-4928-ac90-018c1f05045f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:37.067112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471b8d46-be46-4928-ac90-018c1f05045f admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:37.067394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471b8d46-be46-4928-ac90-018c1f05045f admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:37.067667 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471b8d46-be46-4928-ac90-018c1f05045f admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:37.068299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471b8d46-be46-4928-ac90-018c1f05045f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:37.069924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471b8d46-be46-4928-ac90-018c1f05045f admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:37.080941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1387/2754] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:36 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:37.085067 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cdf680-3f7a-4389-afed-31356a0d0f00 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:37.095771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cdf680-3f7a-4389-afed-31356a0d0f00 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:37.100389 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16487a4b-2b02-4454-ab0a-3fec70873613 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:37.101057 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16487a4b-2b02-4454-ab0a-3fec70873613 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:37.101361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16487a4b-2b02-4454-ab0a-3fec70873613 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:37.101639 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16487a4b-2b02-4454-ab0a-3fec70873613 admin admin] PATH_INFO: `/v3/projects/daebf2d0549c4c6592da9d8a0b878146/users/0954bbf7ae0642b894b21e09a9f2874e/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:37.103563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16487a4b-2b02-4454-ab0a-3fec70873613 admin admin] RBAC: Authorizing `identity:create_grant(project_id=daebf2d0549c4c6592da9d8a0b878146, user_id=0954bbf7ae0642b894b21e09a9f2874e, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:37.106871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16487a4b-2b02-4454-ab0a-3fec70873613 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:37.122410 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1368/2755] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:35:37 2022] PUT /identity/v3/projects/daebf2d0549c4c6592da9d8a0b878146/users/0954bbf7ae0642b894b21e09a9f2874e/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:37.133148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7959d23f-df1c-45f6-bc5e-048b04f7e4c2 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:37.133454 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7959d23f-df1c-45f6-bc5e-048b04f7e4c2 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:37.133712 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7959d23f-df1c-45f6-bc5e-048b04f7e4c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:37.162455 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-7959d23f-df1c-45f6-bc5e-048b04f7e4c2 None None] MFA Rules not processed for user `0954bbf7ae0642b894b21e09a9f2874e`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:37.163666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-7959d23f-df1c-45f6-bc5e-048b04f7e4c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:37.233175 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1388/2756] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:37 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:37.246692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16487a4b-2b02-4454-ab0a-3fec70873613 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:37.248913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-16487a4b-2b02-4454-ab0a-3fec70873613 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:37.287949 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:37.314825 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16487a4b-2b02-4454-ab0a-3fec70873613 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:37.318599 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4dd6bb-9959-40da-8268-0031afb9bf28 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:37.319159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4dd6bb-9959-40da-8268-0031afb9bf28 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:37.319439 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4dd6bb-9959-40da-8268-0031afb9bf28 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:37.319710 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4dd6bb-9959-40da-8268-0031afb9bf28 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:37.329295 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4dd6bb-9959-40da-8268-0031afb9bf28 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:37.330761 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4dd6bb-9959-40da-8268-0031afb9bf28 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:37.342683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1369/2757] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:35:37 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:35:37.673986 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7959d23f-df1c-45f6-bc5e-048b04f7e4c2 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:37.682958 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7959d23f-df1c-45f6-bc5e-048b04f7e4c2 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:37.687549 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d089d1d-a6c1-413d-bf52-1ee917ef5d44 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0954bbf7ae0642b894b21e09a9f2874e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'daebf2d0549c4c6592da9d8a0b878146', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:37.688159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d089d1d-a6c1-413d-bf52-1ee917ef5d44 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:37.688443 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d089d1d-a6c1-413d-bf52-1ee917ef5d44 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:37.688748 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d089d1d-a6c1-413d-bf52-1ee917ef5d44 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:37.689435 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d089d1d-a6c1-413d-bf52-1ee917ef5d44 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:35:37.689741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d089d1d-a6c1-413d-bf52-1ee917ef5d44 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:37.691397 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d089d1d-a6c1-413d-bf52-1ee917ef5d44 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:37.703455 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1389/2758] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:35:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:35:37.707495 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4dd6bb-9959-40da-8268-0031afb9bf28 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:37.717083 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4dd6bb-9959-40da-8268-0031afb9bf28 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:37.721649 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c10c1c-1cce-45fd-9ae8-2047740e1baa tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0954bbf7ae0642b894b21e09a9f2874e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'daebf2d0549c4c6592da9d8a0b878146', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:37.722240 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c10c1c-1cce-45fd-9ae8-2047740e1baa tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:37.722519 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c10c1c-1cce-45fd-9ae8-2047740e1baa tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:37.722795 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c10c1c-1cce-45fd-9ae8-2047740e1baa tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:37.723427 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c10c1c-1cce-45fd-9ae8-2047740e1baa tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:37.725354 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c10c1c-1cce-45fd-9ae8-2047740e1baa tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:37.751925 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1370/2759] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:35:37 2022] POST /identity/v3/projects => generated 355 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:35:37.756023 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d089d1d-a6c1-413d-bf52-1ee917ef5d44 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:37.766301 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d089d1d-a6c1-413d-bf52-1ee917ef5d44 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:37.810872 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:37.827187 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6f14df-75c5-4fc4-aa49-78b768622ea4 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0954bbf7ae0642b894b21e09a9f2874e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'daebf2d0549c4c6592da9d8a0b878146', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:37.827865 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6f14df-75c5-4fc4-aa49-78b768622ea4 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:37.828156 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6f14df-75c5-4fc4-aa49-78b768622ea4 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:37.828433 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6f14df-75c5-4fc4-aa49-78b768622ea4 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:37.829106 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6f14df-75c5-4fc4-aa49-78b768622ea4 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:35:37.829417 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6f14df-75c5-4fc4-aa49-78b768622ea4 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:37.831026 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6f14df-75c5-4fc4-aa49-78b768622ea4 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:37.843004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1390/2760] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:35:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:35:37.878956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c10c1c-1cce-45fd-9ae8-2047740e1baa tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:37.889578 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c10c1c-1cce-45fd-9ae8-2047740e1baa tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:37.894099 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe98730-c250-4afc-adb5-9141cb367798 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0954bbf7ae0642b894b21e09a9f2874e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'daebf2d0549c4c6592da9d8a0b878146', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:37.894717 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe98730-c250-4afc-adb5-9141cb367798 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:37.895001 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe98730-c250-4afc-adb5-9141cb367798 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:37.895276 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe98730-c250-4afc-adb5-9141cb367798 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] PATH_INFO: `/v3/projects/ef14b4d939fb41d8b939802e46c6a139` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:37.896408 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe98730-c250-4afc-adb5-9141cb367798 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] RBAC: Authorizing `identity:delete_project(project_id=ef14b4d939fb41d8b939802e46c6a139)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:37.898353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe98730-c250-4afc-adb5-9141cb367798 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:37.944558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-abe98730-c250-4afc-adb5-9141cb367798 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ef14b4d939fb41d8b939802e46c6a139', 'request_id': 'req-abe98730-c250-4afc-adb5-9141cb367798'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:37.950745 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-abe98730-c250-4afc-adb5-9141cb367798 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ef14b4d939fb41d8b939802e46c6a139', 'request_id': 'req-abe98730-c250-4afc-adb5-9141cb367798'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:37.958134 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-abe98730-c250-4afc-adb5-9141cb367798 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ef14b4d939fb41d8b939802e46c6a139', 'request_id': 'req-abe98730-c250-4afc-adb5-9141cb367798'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:37.969921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-abe98730-c250-4afc-adb5-9141cb367798 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] The token cache is being invalidate because project ef14b4d939fb41d8b939802e46c6a139 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:35:37.970207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-abe98730-c250-4afc-adb5-9141cb367798 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:37.970849 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-abe98730-c250-4afc-adb5-9141cb367798 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:37.972601 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1371/2761] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:35:37 2022] DELETE /identity/v3/projects/ef14b4d939fb41d8b939802e46c6a139 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:38.170482 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6f14df-75c5-4fc4-aa49-78b768622ea4 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:38.172170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-1f6f14df-75c5-4fc4-aa49-78b768622ea4 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:38.236578 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6f14df-75c5-4fc4-aa49-78b768622ea4 tempest-VolumeQuotaClassesTest-911542969 tempest-VolumeQuotaClassesTest-911542969-project-admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:38.291506 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4660f45-d04d-4d44-b617-53f0d3c3b93f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:38.292422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4660f45-d04d-4d44-b617-53f0d3c3b93f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:38.292703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4660f45-d04d-4d44-b617-53f0d3c3b93f admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:38.292973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4660f45-d04d-4d44-b617-53f0d3c3b93f admin admin] PATH_INFO: `/v3/users/dbd17e217fb944ee81a6c39920a439e0` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:38.294146 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4660f45-d04d-4d44-b617-53f0d3c3b93f admin admin] RBAC: Authorizing `identity:delete_user(user_id=dbd17e217fb944ee81a6c39920a439e0)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:38.296045 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4660f45-d04d-4d44-b617-53f0d3c3b93f admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:38.347522 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a4660f45-d04d-4d44-b617-53f0d3c3b93f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dbd17e217fb944ee81a6c39920a439e0'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:38.354064 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a4660f45-d04d-4d44-b617-53f0d3c3b93f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dbd17e217fb944ee81a6c39920a439e0'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:38.354796 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a4660f45-d04d-4d44-b617-53f0d3c3b93f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dbd17e217fb944ee81a6c39920a439e0'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:38.366736 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a4660f45-d04d-4d44-b617-53f0d3c3b93f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dbd17e217fb944ee81a6c39920a439e0'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:38.367403 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a4660f45-d04d-4d44-b617-53f0d3c3b93f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dbd17e217fb944ee81a6c39920a439e0'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:38.380059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1391/2762] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:35:38 2022] DELETE /identity/v3/users/dbd17e217fb944ee81a6c39920a439e0 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:38.384737 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa8ed39-3c85-4e86-bad8-a92bb67e3efd None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:38.384993 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa8ed39-3c85-4e86-bad8-a92bb67e3efd None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:38.385954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa8ed39-3c85-4e86-bad8-a92bb67e3efd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:38.404417 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-faa8ed39-3c85-4e86-bad8-a92bb67e3efd None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:38.405663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-faa8ed39-3c85-4e86-bad8-a92bb67e3efd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:38.466117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1372/2763] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:38.475951 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4660f45-d04d-4d44-b617-53f0d3c3b93f admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:38.477387 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-a4660f45-d04d-4d44-b617-53f0d3c3b93f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:38.484647 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:38.546824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4660f45-d04d-4d44-b617-53f0d3c3b93f admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:38.550725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b148cad-03dc-454d-a668-76793f9312bb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:38.551333 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b148cad-03dc-454d-a668-76793f9312bb service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:38.551610 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b148cad-03dc-454d-a668-76793f9312bb service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:38.551882 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b148cad-03dc-454d-a668-76793f9312bb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:38.561786 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b148cad-03dc-454d-a668-76793f9312bb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:38.563202 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b148cad-03dc-454d-a668-76793f9312bb service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:38.574924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1392/2764] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:35:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:35:38.842033 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa8ed39-3c85-4e86-bad8-a92bb67e3efd None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:38.843626 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-faa8ed39-3c85-4e86-bad8-a92bb67e3efd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:38.857561 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa8ed39-3c85-4e86-bad8-a92bb67e3efd None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:38.863072 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f791d73-9f4b-4420-bd1c-e4f3615b64a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:38.863548 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f791d73-9f4b-4420-bd1c-e4f3615b64a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:38.863839 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f791d73-9f4b-4420-bd1c-e4f3615b64a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:38.864114 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f791d73-9f4b-4420-bd1c-e4f3615b64a0 admin admin] PATH_INFO: `/v3/projects/3918d59d318945a9b3ec45b7704cad66` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:38.865307 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f791d73-9f4b-4420-bd1c-e4f3615b64a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3918d59d318945a9b3ec45b7704cad66)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:38.867269 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f791d73-9f4b-4420-bd1c-e4f3615b64a0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:38.902165 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:38.915321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4f791d73-9f4b-4420-bd1c-e4f3615b64a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3918d59d318945a9b3ec45b7704cad66', 'request_id': 'req-4f791d73-9f4b-4420-bd1c-e4f3615b64a0'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:38.921552 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4f791d73-9f4b-4420-bd1c-e4f3615b64a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3918d59d318945a9b3ec45b7704cad66', 'request_id': 'req-4f791d73-9f4b-4420-bd1c-e4f3615b64a0'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:38.928838 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4f791d73-9f4b-4420-bd1c-e4f3615b64a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3918d59d318945a9b3ec45b7704cad66', 'request_id': 'req-4f791d73-9f4b-4420-bd1c-e4f3615b64a0'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:38.939728 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4f791d73-9f4b-4420-bd1c-e4f3615b64a0 admin admin] The token cache is being invalidate because project 3918d59d318945a9b3ec45b7704cad66 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:35:38.940011 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4f791d73-9f4b-4420-bd1c-e4f3615b64a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:38.940630 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4f791d73-9f4b-4420-bd1c-e4f3615b64a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:38.942133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1373/2765] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:35:38 2022] DELETE /identity/v3/projects/3918d59d318945a9b3ec45b7704cad66 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:38.946117 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b148cad-03dc-454d-a668-76793f9312bb service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:38.956011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b148cad-03dc-454d-a668-76793f9312bb service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:39.014690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8eb4181-e7b3-4515-97f3-73ad324ccca5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:39.015327 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eb4181-e7b3-4515-97f3-73ad324ccca5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:39.015708 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eb4181-e7b3-4515-97f3-73ad324ccca5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:39.015916 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eb4181-e7b3-4515-97f3-73ad324ccca5 admin admin] PATH_INFO: `/v3/users/0954bbf7ae0642b894b21e09a9f2874e` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:39.017324 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8eb4181-e7b3-4515-97f3-73ad324ccca5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0954bbf7ae0642b894b21e09a9f2874e)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:39.019248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8eb4181-e7b3-4515-97f3-73ad324ccca5 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:39.069161 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b8eb4181-e7b3-4515-97f3-73ad324ccca5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0954bbf7ae0642b894b21e09a9f2874e'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:39.077080 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b8eb4181-e7b3-4515-97f3-73ad324ccca5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0954bbf7ae0642b894b21e09a9f2874e'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:39.077759 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b8eb4181-e7b3-4515-97f3-73ad324ccca5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0954bbf7ae0642b894b21e09a9f2874e'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:39.090369 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b8eb4181-e7b3-4515-97f3-73ad324ccca5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0954bbf7ae0642b894b21e09a9f2874e'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:39.091301 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b8eb4181-e7b3-4515-97f3-73ad324ccca5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0954bbf7ae0642b894b21e09a9f2874e'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:39.103203 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1393/2766] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:35:38 2022] DELETE /identity/v3/users/0954bbf7ae0642b894b21e09a9f2874e => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:39.393082 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f791d73-9f4b-4420-bd1c-e4f3615b64a0 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:39.394663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4f791d73-9f4b-4420-bd1c-e4f3615b64a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:39.410284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f791d73-9f4b-4420-bd1c-e4f3615b64a0 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:39.474471 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3be9a94-b75f-4a17-86e2-0556f3970527 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:39.475119 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3be9a94-b75f-4a17-86e2-0556f3970527 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:39.475464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3be9a94-b75f-4a17-86e2-0556f3970527 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:39.475748 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3be9a94-b75f-4a17-86e2-0556f3970527 admin admin] PATH_INFO: `/v3/projects/daebf2d0549c4c6592da9d8a0b878146` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:39.476888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3be9a94-b75f-4a17-86e2-0556f3970527 admin admin] RBAC: Authorizing `identity:delete_project(project_id=daebf2d0549c4c6592da9d8a0b878146)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:39.478864 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3be9a94-b75f-4a17-86e2-0556f3970527 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:39.518140 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8eb4181-e7b3-4515-97f3-73ad324ccca5 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:39.519704 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b8eb4181-e7b3-4515-97f3-73ad324ccca5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:39.528674 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e3be9a94-b75f-4a17-86e2-0556f3970527 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'daebf2d0549c4c6592da9d8a0b878146', 'request_id': 'req-e3be9a94-b75f-4a17-86e2-0556f3970527'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:39.534377 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e3be9a94-b75f-4a17-86e2-0556f3970527 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'daebf2d0549c4c6592da9d8a0b878146', 'request_id': 'req-e3be9a94-b75f-4a17-86e2-0556f3970527'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:39.542701 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e3be9a94-b75f-4a17-86e2-0556f3970527 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'daebf2d0549c4c6592da9d8a0b878146', 'request_id': 'req-e3be9a94-b75f-4a17-86e2-0556f3970527'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:39.553094 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:39.554456 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e3be9a94-b75f-4a17-86e2-0556f3970527 admin admin] The token cache is being invalidate because project daebf2d0549c4c6592da9d8a0b878146 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:35:39.554847 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e3be9a94-b75f-4a17-86e2-0556f3970527 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:39.555584 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e3be9a94-b75f-4a17-86e2-0556f3970527 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:39.557144 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1374/2767] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:35:39 2022] DELETE /identity/v3/projects/daebf2d0549c4c6592da9d8a0b878146 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:39.562508 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a261510f-cee8-423f-b0f9-7e7902ae11b1 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:39.562843 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a261510f-cee8-423f-b0f9-7e7902ae11b1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:39.563126 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a261510f-cee8-423f-b0f9-7e7902ae11b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:39.583175 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-a261510f-cee8-423f-b0f9-7e7902ae11b1 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:39.584403 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a261510f-cee8-423f-b0f9-7e7902ae11b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:39.584746 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8eb4181-e7b3-4515-97f3-73ad324ccca5 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:39.630984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:39.634781 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debd5ad5-a36c-4c5a-974c-789452f71709 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:39.635993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debd5ad5-a36c-4c5a-974c-789452f71709 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:39.636421 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debd5ad5-a36c-4c5a-974c-789452f71709 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:39.636581 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debd5ad5-a36c-4c5a-974c-789452f71709 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:39.638377 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-debd5ad5-a36c-4c5a-974c-789452f71709 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:39.653344 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1375/2768] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:35:39.657709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb80058-f9b8-44fe-831b-24edb2737af1 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:39.658116 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb80058-f9b8-44fe-831b-24edb2737af1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:39.658356 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb80058-f9b8-44fe-831b-24edb2737af1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:39.677707 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-8eb80058-f9b8-44fe-831b-24edb2737af1 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:39.678846 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8eb80058-f9b8-44fe-831b-24edb2737af1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:39.688068 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1376/2769] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:35:39.692393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb80058-f9b8-44fe-831b-24edb2737af1 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:39.714970 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debd5ad5-a36c-4c5a-974c-789452f71709 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:39.715491 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb80058-f9b8-44fe-831b-24edb2737af1 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:39.717023 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debd5ad5-a36c-4c5a-974c-789452f71709 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:39.721692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d46c044-1e2e-43d0-a2be-ba636835460f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:39.722335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d46c044-1e2e-43d0-a2be-ba636835460f admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:39.722611 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d46c044-1e2e-43d0-a2be-ba636835460f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:39.722916 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d46c044-1e2e-43d0-a2be-ba636835460f admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:39.723535 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d46c044-1e2e-43d0-a2be-ba636835460f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:35:39.723853 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d46c044-1e2e-43d0-a2be-ba636835460f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:39.725469 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d46c044-1e2e-43d0-a2be-ba636835460f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:39.727907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1394/2770] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:35:39 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:35:39.739073 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1377/2771] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:35:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:35:39.746102 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debd5ad5-a36c-4c5a-974c-789452f71709 service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:39.756582 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debd5ad5-a36c-4c5a-974c-789452f71709 service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:39.767469 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48a27d8-e48e-472c-bee5-6a01be6b120b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:39.768357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48a27d8-e48e-472c-bee5-6a01be6b120b admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:39.768741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48a27d8-e48e-472c-bee5-6a01be6b120b admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:39.769155 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48a27d8-e48e-472c-bee5-6a01be6b120b admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:39.769901 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48a27d8-e48e-472c-bee5-6a01be6b120b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:39.771907 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48a27d8-e48e-472c-bee5-6a01be6b120b admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:39.806784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1395/2772] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:35:39 2022] POST /identity/v3/projects => generated 385 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:35:39.812089 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d46c044-1e2e-43d0-a2be-ba636835460f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:39.824494 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d46c044-1e2e-43d0-a2be-ba636835460f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:39.908475 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbdaa9d-303e-49f8-ad3a-a2b6487b1964 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:39.909112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbdaa9d-303e-49f8-ad3a-a2b6487b1964 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:39.909390 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbdaa9d-303e-49f8-ad3a-a2b6487b1964 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:39.909656 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbdaa9d-303e-49f8-ad3a-a2b6487b1964 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:39.910304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbdaa9d-303e-49f8-ad3a-a2b6487b1964 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:39.912428 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbdaa9d-303e-49f8-ad3a-a2b6487b1964 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:39.934425 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1378/2773] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:35:39 2022] POST /identity/v3/users => generated 293 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:35:39.938653 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48a27d8-e48e-472c-bee5-6a01be6b120b admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:39.950301 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48a27d8-e48e-472c-bee5-6a01be6b120b admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:39.955715 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedf9b99-bd2e-42b7-a878-edddd504bb8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:39.956487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedf9b99-bd2e-42b7-a878-edddd504bb8e admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:39.956771 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedf9b99-bd2e-42b7-a878-edddd504bb8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:39.957067 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedf9b99-bd2e-42b7-a878-edddd504bb8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:39.957717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedf9b99-bd2e-42b7-a878-edddd504bb8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:39.959311 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedf9b99-bd2e-42b7-a878-edddd504bb8e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:39.970133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1396/2774] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:39 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:39.974366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbdaa9d-303e-49f8-ad3a-a2b6487b1964 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:39.985267 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbdaa9d-303e-49f8-ad3a-a2b6487b1964 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:39.990006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a98c65-415c-473a-ae4d-c7fdb90dba81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:39.990512 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a98c65-415c-473a-ae4d-c7fdb90dba81 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:39.990859 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a98c65-415c-473a-ae4d-c7fdb90dba81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:39.991131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a98c65-415c-473a-ae4d-c7fdb90dba81 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:39.991836 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a98c65-415c-473a-ae4d-c7fdb90dba81 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:39.993475 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a98c65-415c-473a-ae4d-c7fdb90dba81 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:40.004727 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1379/2775] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:39 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:40.009018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedf9b99-bd2e-42b7-a878-edddd504bb8e admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:40.019134 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedf9b99-bd2e-42b7-a878-edddd504bb8e admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:40.025378 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a2dff5-95f3-42eb-aea8-22f07cbbd1a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:40.026041 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a2dff5-95f3-42eb-aea8-22f07cbbd1a3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:40.026326 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a2dff5-95f3-42eb-aea8-22f07cbbd1a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:40.026631 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a2dff5-95f3-42eb-aea8-22f07cbbd1a3 admin admin] PATH_INFO: `/v3/projects/c5fdc44bc43948bea286eac6883003f6/users/7d6742248d394bf7b1141cb7ef37d9ff/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:40.042921 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a2dff5-95f3-42eb-aea8-22f07cbbd1a3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c5fdc44bc43948bea286eac6883003f6, user_id=7d6742248d394bf7b1141cb7ef37d9ff, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:40.045863 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a2dff5-95f3-42eb-aea8-22f07cbbd1a3 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:40.059422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1397/2776] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:35:40 2022] PUT /identity/v3/projects/c5fdc44bc43948bea286eac6883003f6/users/7d6742248d394bf7b1141cb7ef37d9ff/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:35:40.069919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6657dbb5-c11c-4a8e-ab95-621615c0f663 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:40.070209 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6657dbb5-c11c-4a8e-ab95-621615c0f663 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:40.070486 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6657dbb5-c11c-4a8e-ab95-621615c0f663 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:40.099356 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-6657dbb5-c11c-4a8e-ab95-621615c0f663 None None] MFA Rules not processed for user `7d6742248d394bf7b1141cb7ef37d9ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:40.100601 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6657dbb5-c11c-4a8e-ab95-621615c0f663 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:40.175499 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1380/2777] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:40 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:40.179995 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a2dff5-95f3-42eb-aea8-22f07cbbd1a3 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:40.225943 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:40.253436 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a2dff5-95f3-42eb-aea8-22f07cbbd1a3 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:40.258559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313282c4-1d4f-48d8-ab1a-4768db0967c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:40.259160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313282c4-1d4f-48d8-ab1a-4768db0967c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:40.259435 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313282c4-1d4f-48d8-ab1a-4768db0967c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:40.259725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313282c4-1d4f-48d8-ab1a-4768db0967c7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:40.260405 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313282c4-1d4f-48d8-ab1a-4768db0967c7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:40.262357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313282c4-1d4f-48d8-ab1a-4768db0967c7 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:40.290935 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1398/2778] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:35:40 2022] POST /identity/v3/projects => generated 383 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:35:40.295272 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6657dbb5-c11c-4a8e-ab95-621615c0f663 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:40.305827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6657dbb5-c11c-4a8e-ab95-621615c0f663 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:40.344265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:40.365457 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b9307d-5e78-403b-9698-de4849746c46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:40.366075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b9307d-5e78-403b-9698-de4849746c46 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:40.366358 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b9307d-5e78-403b-9698-de4849746c46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:40.366638 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b9307d-5e78-403b-9698-de4849746c46 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:40.367286 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b9307d-5e78-403b-9698-de4849746c46 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:40.369273 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b9307d-5e78-403b-9698-de4849746c46 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:40.390426 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1381/2779] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:35:40 2022] POST /identity/v3/users => generated 298 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:35:40.395000 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313282c4-1d4f-48d8-ab1a-4768db0967c7 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:40.405719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313282c4-1d4f-48d8-ab1a-4768db0967c7 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:40.410239 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00808ede-4b53-40cd-86a1-e8ee99a64b07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:40.410965 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00808ede-4b53-40cd-86a1-e8ee99a64b07 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:40.411242 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00808ede-4b53-40cd-86a1-e8ee99a64b07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:40.411543 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00808ede-4b53-40cd-86a1-e8ee99a64b07 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:40.412170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00808ede-4b53-40cd-86a1-e8ee99a64b07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:40.413780 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00808ede-4b53-40cd-86a1-e8ee99a64b07 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:40.425018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1399/2780] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:40.429873 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b9307d-5e78-403b-9698-de4849746c46 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:40.440542 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b9307d-5e78-403b-9698-de4849746c46 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:40.445435 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052dba31-2c5e-457b-ba0e-eb3e8fad7368 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:40.446035 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052dba31-2c5e-457b-ba0e-eb3e8fad7368 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:40.446314 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052dba31-2c5e-457b-ba0e-eb3e8fad7368 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:40.446592 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052dba31-2c5e-457b-ba0e-eb3e8fad7368 admin admin] PATH_INFO: `/v3/projects/16bf5dbe9d17403581f16ef59d8bfd67/users/a9af70f9d0bb4333a2573b3a3c48a809/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:40.462447 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052dba31-2c5e-457b-ba0e-eb3e8fad7368 admin admin] RBAC: Authorizing `identity:create_grant(project_id=16bf5dbe9d17403581f16ef59d8bfd67, user_id=a9af70f9d0bb4333a2573b3a3c48a809, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:40.465785 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052dba31-2c5e-457b-ba0e-eb3e8fad7368 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:40.480062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1382/2781] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:35:40 2022] PUT /identity/v3/projects/16bf5dbe9d17403581f16ef59d8bfd67/users/a9af70f9d0bb4333a2573b3a3c48a809/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:40.484408 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00808ede-4b53-40cd-86a1-e8ee99a64b07 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:40.494509 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00808ede-4b53-40cd-86a1-e8ee99a64b07 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:40.552503 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e884ea24-7c45-44ff-b7b0-0c0c098dccdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:40.553153 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e884ea24-7c45-44ff-b7b0-0c0c098dccdd admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:40.553435 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e884ea24-7c45-44ff-b7b0-0c0c098dccdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:40.553706 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e884ea24-7c45-44ff-b7b0-0c0c098dccdd admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:40.554345 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e884ea24-7c45-44ff-b7b0-0c0c098dccdd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:40.555927 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e884ea24-7c45-44ff-b7b0-0c0c098dccdd admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:40.578248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1400/2782] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:40 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:40.579953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052dba31-2c5e-457b-ba0e-eb3e8fad7368 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:40.590663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052dba31-2c5e-457b-ba0e-eb3e8fad7368 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:40.595864 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d751802-bc31-46a9-9cdd-6d23c804b52f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:40.596521 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d751802-bc31-46a9-9cdd-6d23c804b52f admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:40.596792 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d751802-bc31-46a9-9cdd-6d23c804b52f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:40.597097 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d751802-bc31-46a9-9cdd-6d23c804b52f admin admin] PATH_INFO: `/v3/projects/16bf5dbe9d17403581f16ef59d8bfd67/users/a9af70f9d0bb4333a2573b3a3c48a809/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:40.599037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d751802-bc31-46a9-9cdd-6d23c804b52f admin admin] RBAC: Authorizing `identity:create_grant(project_id=16bf5dbe9d17403581f16ef59d8bfd67, user_id=a9af70f9d0bb4333a2573b3a3c48a809, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:40.601735 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d751802-bc31-46a9-9cdd-6d23c804b52f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:40.613420 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1383/2783] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:35:40 2022] PUT /identity/v3/projects/16bf5dbe9d17403581f16ef59d8bfd67/users/a9af70f9d0bb4333a2573b3a3c48a809/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:40.625997 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559e777d-584a-4178-be05-64c268496fd5 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:40.626324 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559e777d-584a-4178-be05-64c268496fd5 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:40.626660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559e777d-584a-4178-be05-64c268496fd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:40.655944 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-559e777d-584a-4178-be05-64c268496fd5 None None] MFA Rules not processed for user `a9af70f9d0bb4333a2573b3a3c48a809`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:40.657472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-559e777d-584a-4178-be05-64c268496fd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:40.725919 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1401/2784] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:40 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:40.738416 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d751802-bc31-46a9-9cdd-6d23c804b52f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:40.740117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1d751802-bc31-46a9-9cdd-6d23c804b52f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:40.799542 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d751802-bc31-46a9-9cdd-6d23c804b52f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:40.803214 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757041ec-c6b2-4082-97a3-707627c43b64 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:40.803818 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757041ec-c6b2-4082-97a3-707627c43b64 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:40.804097 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757041ec-c6b2-4082-97a3-707627c43b64 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:40.804369 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757041ec-c6b2-4082-97a3-707627c43b64 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:40.814133 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757041ec-c6b2-4082-97a3-707627c43b64 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:40.815299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757041ec-c6b2-4082-97a3-707627c43b64 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:40.825323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1384/2785] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:35:40 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:35:41.472216 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559e777d-584a-4178-be05-64c268496fd5 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:41.506215 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:41.540753 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559e777d-584a-4178-be05-64c268496fd5 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:41.546593 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4b7f55-9c7d-45de-be80-6723690d1587 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:41.547276 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4b7f55-9c7d-45de-be80-6723690d1587 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:41.547531 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4b7f55-9c7d-45de-be80-6723690d1587 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:41.547798 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4b7f55-9c7d-45de-be80-6723690d1587 admin admin] PATH_INFO: `/v3/users/7d6742248d394bf7b1141cb7ef37d9ff` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:41.549373 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4b7f55-9c7d-45de-be80-6723690d1587 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7d6742248d394bf7b1141cb7ef37d9ff)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:41.551351 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4b7f55-9c7d-45de-be80-6723690d1587 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:41.604580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ba4b7f55-9c7d-45de-be80-6723690d1587 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7d6742248d394bf7b1141cb7ef37d9ff'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:41.613295 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ba4b7f55-9c7d-45de-be80-6723690d1587 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7d6742248d394bf7b1141cb7ef37d9ff'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:41.614047 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ba4b7f55-9c7d-45de-be80-6723690d1587 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7d6742248d394bf7b1141cb7ef37d9ff'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:41.627914 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ba4b7f55-9c7d-45de-be80-6723690d1587 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7d6742248d394bf7b1141cb7ef37d9ff'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:41.628671 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-ba4b7f55-9c7d-45de-be80-6723690d1587 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7d6742248d394bf7b1141cb7ef37d9ff'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:41.642034 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1402/2786] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:35:41 2022] DELETE /identity/v3/users/7d6742248d394bf7b1141cb7ef37d9ff => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:41.648378 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573793ab-bf2d-4555-b6c1-5c886f29a58b None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:41.648920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573793ab-bf2d-4555-b6c1-5c886f29a58b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:41.648920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573793ab-bf2d-4555-b6c1-5c886f29a58b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:41.669734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-573793ab-bf2d-4555-b6c1-5c886f29a58b None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:41.670975 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-573793ab-bf2d-4555-b6c1-5c886f29a58b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:41.733262 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1385/2787] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:41.743501 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4b7f55-9c7d-45de-be80-6723690d1587 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:41.744982 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ba4b7f55-9c7d-45de-be80-6723690d1587 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:41.806265 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4b7f55-9c7d-45de-be80-6723690d1587 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:41.810040 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1878d396-0d0e-48fd-949e-5d7dd4b1f34d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:41.811129 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1878d396-0d0e-48fd-949e-5d7dd4b1f34d service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:41.811129 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1878d396-0d0e-48fd-949e-5d7dd4b1f34d service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:41.811337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1878d396-0d0e-48fd-949e-5d7dd4b1f34d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:41.820868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1878d396-0d0e-48fd-949e-5d7dd4b1f34d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:41.823272 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1878d396-0d0e-48fd-949e-5d7dd4b1f34d service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:41.833472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1403/2788] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:35:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:35:42.133846 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573793ab-bf2d-4555-b6c1-5c886f29a58b None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:42.136180 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-573793ab-bf2d-4555-b6c1-5c886f29a58b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:42.151496 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573793ab-bf2d-4555-b6c1-5c886f29a58b None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:42.156844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6434c08-d6f1-4fa3-9128-933f1f5bd8bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:42.157591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6434c08-d6f1-4fa3-9128-933f1f5bd8bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:42.158016 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6434c08-d6f1-4fa3-9128-933f1f5bd8bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:42.158314 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6434c08-d6f1-4fa3-9128-933f1f5bd8bb admin admin] PATH_INFO: `/v3/projects/c5fdc44bc43948bea286eac6883003f6` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:42.160072 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6434c08-d6f1-4fa3-9128-933f1f5bd8bb admin admin] RBAC: Authorizing `identity:delete_project(project_id=c5fdc44bc43948bea286eac6883003f6)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:42.162795 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6434c08-d6f1-4fa3-9128-933f1f5bd8bb admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:42.205391 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=93,keystone:INSERT=5,keystone:DELETE=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:42.223955 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d6434c08-d6f1-4fa3-9128-933f1f5bd8bb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c5fdc44bc43948bea286eac6883003f6', 'request_id': 'req-d6434c08-d6f1-4fa3-9128-933f1f5bd8bb'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:42.238247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d6434c08-d6f1-4fa3-9128-933f1f5bd8bb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c5fdc44bc43948bea286eac6883003f6', 'request_id': 'req-d6434c08-d6f1-4fa3-9128-933f1f5bd8bb'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:42.255335 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d6434c08-d6f1-4fa3-9128-933f1f5bd8bb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c5fdc44bc43948bea286eac6883003f6', 'request_id': 'req-d6434c08-d6f1-4fa3-9128-933f1f5bd8bb'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:42.269153 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d6434c08-d6f1-4fa3-9128-933f1f5bd8bb admin admin] The token cache is being invalidate because project c5fdc44bc43948bea286eac6883003f6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:35:42.269531 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d6434c08-d6f1-4fa3-9128-933f1f5bd8bb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:42.270247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d6434c08-d6f1-4fa3-9128-933f1f5bd8bb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:42.272544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1386/2789] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:35:42 2022] DELETE /identity/v3/projects/c5fdc44bc43948bea286eac6883003f6 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:42.276738 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1878d396-0d0e-48fd-949e-5d7dd4b1f34d service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:42.287889 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1878d396-0d0e-48fd-949e-5d7dd4b1f34d service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:42.356227 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:42.367625 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa110101-2d3b-467f-8964-f012c419541a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:42.368380 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa110101-2d3b-467f-8964-f012c419541a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:42.368738 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa110101-2d3b-467f-8964-f012c419541a admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:42.369143 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa110101-2d3b-467f-8964-f012c419541a admin admin] PATH_INFO: `/v3/users/a9af70f9d0bb4333a2573b3a3c48a809` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:42.370934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa110101-2d3b-467f-8964-f012c419541a admin admin] RBAC: Authorizing `identity:delete_user(user_id=a9af70f9d0bb4333a2573b3a3c48a809)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:42.372954 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa110101-2d3b-467f-8964-f012c419541a admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:42.419908 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-fa110101-2d3b-467f-8964-f012c419541a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a9af70f9d0bb4333a2573b3a3c48a809'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:42.428092 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-fa110101-2d3b-467f-8964-f012c419541a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a9af70f9d0bb4333a2573b3a3c48a809'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:42.428868 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-fa110101-2d3b-467f-8964-f012c419541a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a9af70f9d0bb4333a2573b3a3c48a809'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:42.442390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-fa110101-2d3b-467f-8964-f012c419541a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a9af70f9d0bb4333a2573b3a3c48a809'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:42.443603 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-fa110101-2d3b-467f-8964-f012c419541a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a9af70f9d0bb4333a2573b3a3c48a809'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:42.455814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1404/2790] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:35:42 2022] DELETE /identity/v3/users/a9af70f9d0bb4333a2573b3a3c48a809 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:42.735310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6434c08-d6f1-4fa3-9128-933f1f5bd8bb admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:42.737337 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d6434c08-d6f1-4fa3-9128-933f1f5bd8bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:42.757510 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6434c08-d6f1-4fa3-9128-933f1f5bd8bb admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:42.818630 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4f04dd-7b01-43d5-aedd-6648f950fdb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:42.819282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4f04dd-7b01-43d5-aedd-6648f950fdb6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:42.819563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4f04dd-7b01-43d5-aedd-6648f950fdb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:42.819837 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4f04dd-7b01-43d5-aedd-6648f950fdb6 admin admin] PATH_INFO: `/v3/projects/16bf5dbe9d17403581f16ef59d8bfd67` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:42.820971 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4f04dd-7b01-43d5-aedd-6648f950fdb6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=16bf5dbe9d17403581f16ef59d8bfd67)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:42.823017 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4f04dd-7b01-43d5-aedd-6648f950fdb6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:42.874343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4b4f04dd-7b01-43d5-aedd-6648f950fdb6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '16bf5dbe9d17403581f16ef59d8bfd67', 'request_id': 'req-4b4f04dd-7b01-43d5-aedd-6648f950fdb6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:42.880369 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4b4f04dd-7b01-43d5-aedd-6648f950fdb6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '16bf5dbe9d17403581f16ef59d8bfd67', 'request_id': 'req-4b4f04dd-7b01-43d5-aedd-6648f950fdb6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:42.888509 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4b4f04dd-7b01-43d5-aedd-6648f950fdb6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '16bf5dbe9d17403581f16ef59d8bfd67', 'request_id': 'req-4b4f04dd-7b01-43d5-aedd-6648f950fdb6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:42.900247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4b4f04dd-7b01-43d5-aedd-6648f950fdb6 admin admin] The token cache is being invalidate because project 16bf5dbe9d17403581f16ef59d8bfd67 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:35:42.901031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4b4f04dd-7b01-43d5-aedd-6648f950fdb6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:42.901753 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4b4f04dd-7b01-43d5-aedd-6648f950fdb6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:42.903948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1387/2791] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:35:42 2022] DELETE /identity/v3/projects/16bf5dbe9d17403581f16ef59d8bfd67 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:42.908869 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4836a29-7332-4576-976f-fe41ec0d8c9a None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:42.909358 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4836a29-7332-4576-976f-fe41ec0d8c9a None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:42.909444 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4836a29-7332-4576-976f-fe41ec0d8c9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:42.930910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-c4836a29-7332-4576-976f-fe41ec0d8c9a None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:42.932157 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c4836a29-7332-4576-976f-fe41ec0d8c9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:42.994937 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1405/2792] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:43.003022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e15e3c1-9a21-4d40-b7aa-ce57327f247e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:43.003310 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e15e3c1-9a21-4d40-b7aa-ce57327f247e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:43.003613 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e15e3c1-9a21-4d40-b7aa-ce57327f247e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:43.028728 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-9e15e3c1-9a21-4d40-b7aa-ce57327f247e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:43.030272 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9e15e3c1-9a21-4d40-b7aa-ce57327f247e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:43.039394 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1388/2793] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:43.043779 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4836a29-7332-4576-976f-fe41ec0d8c9a None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:43.053570 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4836a29-7332-4576-976f-fe41ec0d8c9a None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:43.058718 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c73c3fc-7831-48b2-8f38-a534a0e1dd55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:43.059322 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c73c3fc-7831-48b2-8f38-a534a0e1dd55 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:43.059599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c73c3fc-7831-48b2-8f38-a534a0e1dd55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:43.059870 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c73c3fc-7831-48b2-8f38-a534a0e1dd55 admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:43.060519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c73c3fc-7831-48b2-8f38-a534a0e1dd55 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:35:43.060822 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c73c3fc-7831-48b2-8f38-a534a0e1dd55 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:43.062446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c73c3fc-7831-48b2-8f38-a534a0e1dd55 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:43.075662 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1406/2794] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:35:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:35:43.080015 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e15e3c1-9a21-4d40-b7aa-ce57327f247e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:43.090136 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e15e3c1-9a21-4d40-b7aa-ce57327f247e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:43.094888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b49abf-b99b-4059-97f6-983ce5b8ee49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:43.095479 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b49abf-b99b-4059-97f6-983ce5b8ee49 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:43.095755 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b49abf-b99b-4059-97f6-983ce5b8ee49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:43.096026 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b49abf-b99b-4059-97f6-983ce5b8ee49 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:43.096663 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b49abf-b99b-4059-97f6-983ce5b8ee49 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:43.098598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b49abf-b99b-4059-97f6-983ce5b8ee49 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:43.126018 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1389/2795] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:35:43 2022] POST /identity/v3/projects => generated 385 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:35:43.130237 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c73c3fc-7831-48b2-8f38-a534a0e1dd55 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:43.140750 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c73c3fc-7831-48b2-8f38-a534a0e1dd55 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:43.196533 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5018bf-fa2e-451e-8309-4f4a73ec9c97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:43.197158 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5018bf-fa2e-451e-8309-4f4a73ec9c97 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:43.197952 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5018bf-fa2e-451e-8309-4f4a73ec9c97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:43.198223 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5018bf-fa2e-451e-8309-4f4a73ec9c97 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:43.198862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5018bf-fa2e-451e-8309-4f4a73ec9c97 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:43.200811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5018bf-fa2e-451e-8309-4f4a73ec9c97 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:43.208556 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:43.224524 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1407/2796] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:35:43 2022] POST /identity/v3/users => generated 293 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:35:43.228632 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b49abf-b99b-4059-97f6-983ce5b8ee49 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:43.239151 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b49abf-b99b-4059-97f6-983ce5b8ee49 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:43.243824 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21d613b-2f43-41ef-9b99-3ec32931aa10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:43.244463 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21d613b-2f43-41ef-9b99-3ec32931aa10 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:43.244735 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21d613b-2f43-41ef-9b99-3ec32931aa10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:43.245002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21d613b-2f43-41ef-9b99-3ec32931aa10 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:43.245716 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21d613b-2f43-41ef-9b99-3ec32931aa10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:43.247337 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21d613b-2f43-41ef-9b99-3ec32931aa10 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:43.258590 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1390/2797] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:43 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:43.262395 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5018bf-fa2e-451e-8309-4f4a73ec9c97 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:43.272336 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5018bf-fa2e-451e-8309-4f4a73ec9c97 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:43.277617 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8f832b-ebfc-4e63-8fdb-43d9d728b38e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:43.278211 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8f832b-ebfc-4e63-8fdb-43d9d728b38e admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:43.278489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8f832b-ebfc-4e63-8fdb-43d9d728b38e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:43.278758 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8f832b-ebfc-4e63-8fdb-43d9d728b38e admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:43.279400 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8f832b-ebfc-4e63-8fdb-43d9d728b38e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:43.280971 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8f832b-ebfc-4e63-8fdb-43d9d728b38e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:43.292013 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1408/2798] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:43 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:43.296618 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21d613b-2f43-41ef-9b99-3ec32931aa10 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:43.307174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21d613b-2f43-41ef-9b99-3ec32931aa10 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:43.314549 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6edc0a-faed-400e-babc-1b76e44b5242 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:43.315425 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6edc0a-faed-400e-babc-1b76e44b5242 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:43.315952 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6edc0a-faed-400e-babc-1b76e44b5242 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:43.316408 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6edc0a-faed-400e-babc-1b76e44b5242 admin admin] PATH_INFO: `/v3/projects/43389c301db64a48b6aff1246ae14b06/users/f77f99cfb6ca467082dd8dc363dff067/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:43.320633 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:43.335009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6edc0a-faed-400e-babc-1b76e44b5242 admin admin] RBAC: Authorizing `identity:create_grant(project_id=43389c301db64a48b6aff1246ae14b06, user_id=f77f99cfb6ca467082dd8dc363dff067, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:43.337818 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6edc0a-faed-400e-babc-1b76e44b5242 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:43.353393 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1391/2799] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:35:43 2022] PUT /identity/v3/projects/43389c301db64a48b6aff1246ae14b06/users/f77f99cfb6ca467082dd8dc363dff067/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:43.364208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eae42ec-b8b2-4005-8830-10a79617eb72 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:43.364491 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eae42ec-b8b2-4005-8830-10a79617eb72 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:43.364834 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eae42ec-b8b2-4005-8830-10a79617eb72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:43.395926 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-0eae42ec-b8b2-4005-8830-10a79617eb72 None None] MFA Rules not processed for user `f77f99cfb6ca467082dd8dc363dff067`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:43.397179 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-0eae42ec-b8b2-4005-8830-10a79617eb72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:43.466595 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1409/2800] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:43 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:43.477399 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6edc0a-faed-400e-babc-1b76e44b5242 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:43.479113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5f6edc0a-faed-400e-babc-1b76e44b5242 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:43.536094 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6edc0a-faed-400e-babc-1b76e44b5242 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:43.539670 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee63ec23-adf4-4d92-8513-4f7831e76ba2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:43.540269 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee63ec23-adf4-4d92-8513-4f7831e76ba2 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:43.540546 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee63ec23-adf4-4d92-8513-4f7831e76ba2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:43.540817 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee63ec23-adf4-4d92-8513-4f7831e76ba2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:43.548863 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee63ec23-adf4-4d92-8513-4f7831e76ba2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:43.550322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee63ec23-adf4-4d92-8513-4f7831e76ba2 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:43.559356 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1392/2801] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:35:43 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:35:52.383577 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eae42ec-b8b2-4005-8830-10a79617eb72 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:52.445105 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eae42ec-b8b2-4005-8830-10a79617eb72 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:52.450189 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a173590-f855-4020-ad87-be7bb5510ab7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:52.450817 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a173590-f855-4020-ad87-be7bb5510ab7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:52.451094 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a173590-f855-4020-ad87-be7bb5510ab7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:52.451366 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a173590-f855-4020-ad87-be7bb5510ab7 admin admin] PATH_INFO: `/v3/users/f77f99cfb6ca467082dd8dc363dff067` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:52.452633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a173590-f855-4020-ad87-be7bb5510ab7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f77f99cfb6ca467082dd8dc363dff067)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:52.454645 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a173590-f855-4020-ad87-be7bb5510ab7 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:52.502175 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-3a173590-f855-4020-ad87-be7bb5510ab7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f77f99cfb6ca467082dd8dc363dff067'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:52.509580 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-3a173590-f855-4020-ad87-be7bb5510ab7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f77f99cfb6ca467082dd8dc363dff067'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:52.510678 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-3a173590-f855-4020-ad87-be7bb5510ab7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f77f99cfb6ca467082dd8dc363dff067'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:52.522273 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:INSERT=4,keystone:SELECT=86,keystone:DELETE=10 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:52.523480 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-3a173590-f855-4020-ad87-be7bb5510ab7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f77f99cfb6ca467082dd8dc363dff067'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:52.524459 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-3a173590-f855-4020-ad87-be7bb5510ab7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f77f99cfb6ca467082dd8dc363dff067'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:52.539110 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1410/2802] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:35:52 2022] DELETE /identity/v3/users/f77f99cfb6ca467082dd8dc363dff067 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:52.543580 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d28861c-a318-43ea-a51f-8063d893b077 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:52.544059 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d28861c-a318-43ea-a51f-8063d893b077 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:52.544150 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d28861c-a318-43ea-a51f-8063d893b077 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:52.563598 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-0d28861c-a318-43ea-a51f-8063d893b077 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:52.564809 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0d28861c-a318-43ea-a51f-8063d893b077 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:52.625022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1393/2803] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:52.634813 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a173590-f855-4020-ad87-be7bb5510ab7 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:52.636960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-3a173590-f855-4020-ad87-be7bb5510ab7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:52.695287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a173590-f855-4020-ad87-be7bb5510ab7 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:52.699439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a130d57-6c8e-4ffe-839c-44bf6f3e81b2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:52.700042 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a130d57-6c8e-4ffe-839c-44bf6f3e81b2 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:52.700347 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a130d57-6c8e-4ffe-839c-44bf6f3e81b2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:52.700619 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a130d57-6c8e-4ffe-839c-44bf6f3e81b2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:52.710165 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a130d57-6c8e-4ffe-839c-44bf6f3e81b2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:52.711673 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a130d57-6c8e-4ffe-839c-44bf6f3e81b2 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:52.722159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1411/2804] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:35:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:35:53.002781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d28861c-a318-43ea-a51f-8063d893b077 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:53.004279 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0d28861c-a318-43ea-a51f-8063d893b077 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:53.032257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d28861c-a318-43ea-a51f-8063d893b077 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:53.040904 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca0c916-a6e2-488d-a1ca-1af95e2a27e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:53.042157 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca0c916-a6e2-488d-a1ca-1af95e2a27e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:53.042445 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca0c916-a6e2-488d-a1ca-1af95e2a27e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:53.042715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca0c916-a6e2-488d-a1ca-1af95e2a27e6 admin admin] PATH_INFO: `/v3/projects/43389c301db64a48b6aff1246ae14b06` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:53.044513 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca0c916-a6e2-488d-a1ca-1af95e2a27e6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=43389c301db64a48b6aff1246ae14b06)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:53.047958 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca0c916-a6e2-488d-a1ca-1af95e2a27e6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:53.110373 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-fca0c916-a6e2-488d-a1ca-1af95e2a27e6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '43389c301db64a48b6aff1246ae14b06', 'request_id': 'req-fca0c916-a6e2-488d-a1ca-1af95e2a27e6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:53.117913 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-fca0c916-a6e2-488d-a1ca-1af95e2a27e6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '43389c301db64a48b6aff1246ae14b06', 'request_id': 'req-fca0c916-a6e2-488d-a1ca-1af95e2a27e6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:53.126483 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-fca0c916-a6e2-488d-a1ca-1af95e2a27e6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '43389c301db64a48b6aff1246ae14b06', 'request_id': 'req-fca0c916-a6e2-488d-a1ca-1af95e2a27e6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:53.139072 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-fca0c916-a6e2-488d-a1ca-1af95e2a27e6 admin admin] The token cache is being invalidate because project 43389c301db64a48b6aff1246ae14b06 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:35:53.139493 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-fca0c916-a6e2-488d-a1ca-1af95e2a27e6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:53.140230 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-fca0c916-a6e2-488d-a1ca-1af95e2a27e6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:53.141871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1394/2805] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:35:53 2022] DELETE /identity/v3/projects/43389c301db64a48b6aff1246ae14b06 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:54.270528 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a130d57-6c8e-4ffe-839c-44bf6f3e81b2 service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:54.271926 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-0a130d57-6c8e-4ffe-839c-44bf6f3e81b2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:54.283816 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a130d57-6c8e-4ffe-839c-44bf6f3e81b2 service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:54.330889 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff331c8f-ecb0-489b-9979-e71c4d019b46 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:54.331487 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff331c8f-ecb0-489b-9979-e71c4d019b46 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:54.331763 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff331c8f-ecb0-489b-9979-e71c4d019b46 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:54.332033 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff331c8f-ecb0-489b-9979-e71c4d019b46 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:54.333760 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ff331c8f-ecb0-489b-9979-e71c4d019b46 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:54.397699 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff331c8f-ecb0-489b-9979-e71c4d019b46 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:54.399138 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff331c8f-ecb0-489b-9979-e71c4d019b46 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:54.409599 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1412/2806] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:35:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:35:57.508030 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca0c916-a6e2-488d-a1ca-1af95e2a27e6 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:57.516638 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fca0c916-a6e2-488d-a1ca-1af95e2a27e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:57.530268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca0c916-a6e2-488d-a1ca-1af95e2a27e6 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:57.534752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6a15f0-c845-43aa-92b4-7d7fa8e88007 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:57.535371 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6a15f0-c845-43aa-92b4-7d7fa8e88007 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:57.535649 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6a15f0-c845-43aa-92b4-7d7fa8e88007 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:57.535919 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6a15f0-c845-43aa-92b4-7d7fa8e88007 admin admin] PATH_INFO: `/v3/users/b05dcf62adc949c79b37c92bb3e09e8e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:57.537004 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6a15f0-c845-43aa-92b4-7d7fa8e88007 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b05dcf62adc949c79b37c92bb3e09e8e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:57.539069 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6a15f0-c845-43aa-92b4-7d7fa8e88007 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:57.543676 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:57.587419 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3f6a15f0-c845-43aa-92b4-7d7fa8e88007 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b05dcf62adc949c79b37c92bb3e09e8e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:57.599907 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3f6a15f0-c845-43aa-92b4-7d7fa8e88007 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b05dcf62adc949c79b37c92bb3e09e8e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:57.606821 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3f6a15f0-c845-43aa-92b4-7d7fa8e88007 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b05dcf62adc949c79b37c92bb3e09e8e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:57.614353 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3f6a15f0-c845-43aa-92b4-7d7fa8e88007 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b05dcf62adc949c79b37c92bb3e09e8e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:57.615008 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3f6a15f0-c845-43aa-92b4-7d7fa8e88007 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b05dcf62adc949c79b37c92bb3e09e8e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:57.621401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1395/2807] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:35:57 2022] DELETE /identity/v3/users/b05dcf62adc949c79b37c92bb3e09e8e => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:57.787248 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff331c8f-ecb0-489b-9979-e71c4d019b46 service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:57.799669 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff331c8f-ecb0-489b-9979-e71c4d019b46 service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:57.840240 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=100 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:57.859253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a2f262-cb5e-4098-b83f-1943da5ea21c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:57.859877 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a2f262-cb5e-4098-b83f-1943da5ea21c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:57.860179 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a2f262-cb5e-4098-b83f-1943da5ea21c admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:57.860448 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a2f262-cb5e-4098-b83f-1943da5ea21c admin admin] PATH_INFO: `/v3/projects/8afb4049582c4beaa9defa2fe20c8b52` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:57.861575 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a2f262-cb5e-4098-b83f-1943da5ea21c admin admin] RBAC: Authorizing `identity:delete_project(project_id=8afb4049582c4beaa9defa2fe20c8b52)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:57.863633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a2f262-cb5e-4098-b83f-1943da5ea21c admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:57.913804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d0a2f262-cb5e-4098-b83f-1943da5ea21c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8afb4049582c4beaa9defa2fe20c8b52', 'request_id': 'req-d0a2f262-cb5e-4098-b83f-1943da5ea21c'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:57.919362 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d0a2f262-cb5e-4098-b83f-1943da5ea21c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8afb4049582c4beaa9defa2fe20c8b52', 'request_id': 'req-d0a2f262-cb5e-4098-b83f-1943da5ea21c'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:57.928508 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d0a2f262-cb5e-4098-b83f-1943da5ea21c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8afb4049582c4beaa9defa2fe20c8b52', 'request_id': 'req-d0a2f262-cb5e-4098-b83f-1943da5ea21c'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:57.941116 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d0a2f262-cb5e-4098-b83f-1943da5ea21c admin admin] The token cache is being invalidate because project 8afb4049582c4beaa9defa2fe20c8b52 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:35:57.941414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d0a2f262-cb5e-4098-b83f-1943da5ea21c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:57.942069 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d0a2f262-cb5e-4098-b83f-1943da5ea21c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:35:57.943804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1413/2808] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:35:57 2022] DELETE /identity/v3/projects/8afb4049582c4beaa9defa2fe20c8b52 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:35:57.949062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a504449b-02f5-4886-9e3a-f47bedb5af97 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:57.949421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a504449b-02f5-4886-9e3a-f47bedb5af97 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:57.949778 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a504449b-02f5-4886-9e3a-f47bedb5af97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:57.969322 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-a504449b-02f5-4886-9e3a-f47bedb5af97 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:57.970619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a504449b-02f5-4886-9e3a-f47bedb5af97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:58.033564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1396/2809] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:58.042772 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ff4eb4-8875-49f9-ade9-7d2ae8de3e8e None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:58.043060 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ff4eb4-8875-49f9-ade9-7d2ae8de3e8e None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:58.043359 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ff4eb4-8875-49f9-ade9-7d2ae8de3e8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:58.064115 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-12ff4eb4-8875-49f9-ade9-7d2ae8de3e8e None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:58.065357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-12ff4eb4-8875-49f9-ade9-7d2ae8de3e8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:58.073391 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1414/2810] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:58.077551 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a504449b-02f5-4886-9e3a-f47bedb5af97 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:58.087261 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a504449b-02f5-4886-9e3a-f47bedb5af97 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:58.091941 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc60849-bc6d-4907-a5c3-467692e7d97e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:58.092539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc60849-bc6d-4907-a5c3-467692e7d97e admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:58.092844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc60849-bc6d-4907-a5c3-467692e7d97e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:58.093147 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc60849-bc6d-4907-a5c3-467692e7d97e admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:58.093795 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc60849-bc6d-4907-a5c3-467692e7d97e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:35:58.094097 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc60849-bc6d-4907-a5c3-467692e7d97e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:58.095693 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc60849-bc6d-4907-a5c3-467692e7d97e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:58.107670 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1397/2811] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:35:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:35:58.112308 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ff4eb4-8875-49f9-ade9-7d2ae8de3e8e None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:58.122553 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ff4eb4-8875-49f9-ade9-7d2ae8de3e8e None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:58.127190 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d770a1b8-9c47-41fd-b3c6-0f8b8dbe7497 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:58.127789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d770a1b8-9c47-41fd-b3c6-0f8b8dbe7497 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:58.128069 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d770a1b8-9c47-41fd-b3c6-0f8b8dbe7497 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:58.128363 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d770a1b8-9c47-41fd-b3c6-0f8b8dbe7497 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:58.129000 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d770a1b8-9c47-41fd-b3c6-0f8b8dbe7497 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:58.130918 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d770a1b8-9c47-41fd-b3c6-0f8b8dbe7497 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:58.159874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1415/2812] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:35:58 2022] POST /identity/v3/projects => generated 383 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:35:58.164060 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc60849-bc6d-4907-a5c3-467692e7d97e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:58.175764 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc60849-bc6d-4907-a5c3-467692e7d97e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:58.232290 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae38147a-54e4-4bb8-9a62-13fee5e589dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:58.232907 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae38147a-54e4-4bb8-9a62-13fee5e589dd admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:58.233481 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae38147a-54e4-4bb8-9a62-13fee5e589dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:58.233753 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae38147a-54e4-4bb8-9a62-13fee5e589dd admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:58.234394 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae38147a-54e4-4bb8-9a62-13fee5e589dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:58.236341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae38147a-54e4-4bb8-9a62-13fee5e589dd admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:58.253628 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:35:58.259980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1398/2813] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:35:58 2022] POST /identity/v3/users => generated 292 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:35:58.264775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d770a1b8-9c47-41fd-b3c6-0f8b8dbe7497 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:58.275521 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d770a1b8-9c47-41fd-b3c6-0f8b8dbe7497 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:58.280270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61d7007-9b78-4370-9064-d11196b94deb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:58.280866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61d7007-9b78-4370-9064-d11196b94deb admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:58.281173 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61d7007-9b78-4370-9064-d11196b94deb admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:58.281446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61d7007-9b78-4370-9064-d11196b94deb admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:58.282098 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61d7007-9b78-4370-9064-d11196b94deb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:58.283676 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61d7007-9b78-4370-9064-d11196b94deb admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:58.294166 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1416/2814] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:58.298153 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae38147a-54e4-4bb8-9a62-13fee5e589dd admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:58.308889 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae38147a-54e4-4bb8-9a62-13fee5e589dd admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:58.313574 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5346fffe-2b2a-4bc8-82bb-7ba97c3056c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:58.314168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5346fffe-2b2a-4bc8-82bb-7ba97c3056c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:58.314444 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5346fffe-2b2a-4bc8-82bb-7ba97c3056c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:58.314710 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5346fffe-2b2a-4bc8-82bb-7ba97c3056c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:58.315349 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5346fffe-2b2a-4bc8-82bb-7ba97c3056c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:58.316963 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5346fffe-2b2a-4bc8-82bb-7ba97c3056c5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:58.330037 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1399/2815] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:35:58 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:35:58.334301 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61d7007-9b78-4370-9064-d11196b94deb admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:58.345645 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61d7007-9b78-4370-9064-d11196b94deb admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:58.350266 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11b51be-d589-4cef-91ba-1c4b16a7b388 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:58.350908 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11b51be-d589-4cef-91ba-1c4b16a7b388 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:58.351186 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11b51be-d589-4cef-91ba-1c4b16a7b388 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:58.351453 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11b51be-d589-4cef-91ba-1c4b16a7b388 admin admin] PATH_INFO: `/v3/projects/c561fdf35e454833b521f54043c49035/users/8df68cb66f6d4369bba0eda189b8e64d/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:58.366679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11b51be-d589-4cef-91ba-1c4b16a7b388 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c561fdf35e454833b521f54043c49035, user_id=8df68cb66f6d4369bba0eda189b8e64d, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:58.369346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11b51be-d589-4cef-91ba-1c4b16a7b388 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:58.382330 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1417/2816] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:35:58 2022] PUT /identity/v3/projects/c561fdf35e454833b521f54043c49035/users/8df68cb66f6d4369bba0eda189b8e64d/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:35:58.387504 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac904589-d27d-47b5-8619-f731eac8dfc6 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:58.387783 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac904589-d27d-47b5-8619-f731eac8dfc6 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:58.388055 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac904589-d27d-47b5-8619-f731eac8dfc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:58.406991 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ac904589-d27d-47b5-8619-f731eac8dfc6 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:35:58.408189 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ac904589-d27d-47b5-8619-f731eac8dfc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:58.469336 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1400/2817] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:35:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:35:58.479964 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11b51be-d589-4cef-91ba-1c4b16a7b388 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:35:58.481760 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-e11b51be-d589-4cef-91ba-1c4b16a7b388 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:35:58.537534 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11b51be-d589-4cef-91ba-1c4b16a7b388 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:35:58.541684 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661deb42-e95a-4a40-83c5-a7d602cf49fc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:35:58.542291 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661deb42-e95a-4a40-83c5-a7d602cf49fc service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:35:58.542568 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661deb42-e95a-4a40-83c5-a7d602cf49fc service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:35:58.542838 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661deb42-e95a-4a40-83c5-a7d602cf49fc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:35:58.552163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661deb42-e95a-4a40-83c5-a7d602cf49fc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:35:58.553623 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661deb42-e95a-4a40-83c5-a7d602cf49fc service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:35:58.562925 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1418/2818] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:35:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:36:02.562270 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cba72f-5a60-4932-b4d8-a9c76774435b None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:36:02.562822 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cba72f-5a60-4932-b4d8-a9c76774435b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:36:02.562822 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cba72f-5a60-4932-b4d8-a9c76774435b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:36:02.590538 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-e7cba72f-5a60-4932-b4d8-a9c76774435b None None] MFA Rules not processed for user `8df68cb66f6d4369bba0eda189b8e64d`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:36:02.591754 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e7cba72f-5a60-4932-b4d8-a9c76774435b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:36:02.656986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1401/2819] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:36:02 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:36:02.671541 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661deb42-e95a-4a40-83c5-a7d602cf49fc service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:36:02.673159 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-661deb42-e95a-4a40-83c5-a7d602cf49fc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:36:02.685508 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:36:02.752279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661deb42-e95a-4a40-83c5-a7d602cf49fc service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:36:02.758372 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e89db18-b690-456a-b8ba-880be7794b11 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:36:02.759068 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e89db18-b690-456a-b8ba-880be7794b11 service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:36:02.759422 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e89db18-b690-456a-b8ba-880be7794b11 service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:36:02.759770 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e89db18-b690-456a-b8ba-880be7794b11 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:36:02.768856 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e89db18-b690-456a-b8ba-880be7794b11 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:36:02.770244 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e89db18-b690-456a-b8ba-880be7794b11 service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:36:02.780091 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1419/2820] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:36:02 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:36:04.315185 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49210c3f-33ce-41cd-997d-05b77a49fbe3 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:36:04.315619 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49210c3f-33ce-41cd-997d-05b77a49fbe3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:36:04.315830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49210c3f-33ce-41cd-997d-05b77a49fbe3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:36:04.337312 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-49210c3f-33ce-41cd-997d-05b77a49fbe3 None None] MFA Rules not processed for user `8df68cb66f6d4369bba0eda189b8e64d`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:36:04.339079 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-49210c3f-33ce-41cd-997d-05b77a49fbe3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:36:04.346844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1402/2821] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:36:04 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:36:12.775945 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=30 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:36:13.068644 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e89db18-b690-456a-b8ba-880be7794b11 service nova] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:36:13.077811 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e89db18-b690-456a-b8ba-880be7794b11 service nova] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:36:13.081322 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42132fb0-fa27-4602-8a39-1b893c28b44a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:36:13.081880 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42132fb0-fa27-4602-8a39-1b893c28b44a service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:36:13.082152 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42132fb0-fa27-4602-8a39-1b893c28b44a service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:36:13.082415 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42132fb0-fa27-4602-8a39-1b893c28b44a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:36:13.084148 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-42132fb0-fa27-4602-8a39-1b893c28b44a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:36:13.097154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42132fb0-fa27-4602-8a39-1b893c28b44a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:36:13.098559 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42132fb0-fa27-4602-8a39-1b893c28b44a service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:36:13.111294 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1420/2822] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:36:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:36:14.331183 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=67 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:36:15.455330 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49210c3f-33ce-41cd-997d-05b77a49fbe3 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:36:15.465857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49210c3f-33ce-41cd-997d-05b77a49fbe3 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:36:15.469525 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767a4d2c-e5dc-4421-aaf1-c3e561e1bf0a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:36:15.470120 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767a4d2c-e5dc-4421-aaf1-c3e561e1bf0a service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:36:15.470388 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767a4d2c-e5dc-4421-aaf1-c3e561e1bf0a service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:36:15.470659 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767a4d2c-e5dc-4421-aaf1-c3e561e1bf0a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:36:15.472550 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-767a4d2c-e5dc-4421-aaf1-c3e561e1bf0a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:36:15.490293 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767a4d2c-e5dc-4421-aaf1-c3e561e1bf0a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:36:15.491739 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767a4d2c-e5dc-4421-aaf1-c3e561e1bf0a service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:36:15.503617 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1403/2823] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:36:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:36:23.104824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=6 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:36:25.498400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=6 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:37:12.406212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7858630b-81c4-4f88-8269-4c21c7ed8be9 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:37:12.406729 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7858630b-81c4-4f88-8269-4c21c7ed8be9 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:37:12.406844 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7858630b-81c4-4f88-8269-4c21c7ed8be9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:37:12.425588 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-7858630b-81c4-4f88-8269-4c21c7ed8be9 None None] MFA Rules not processed for user `8df68cb66f6d4369bba0eda189b8e64d`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:37:12.426789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-7858630b-81c4-4f88-8269-4c21c7ed8be9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:37:12.433611 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1421/2824] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:37:12 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:37:15.190316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767a4d2c-e5dc-4421-aaf1-c3e561e1bf0a service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:37:15.191884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-767a4d2c-e5dc-4421-aaf1-c3e561e1bf0a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:37:15.203127 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767a4d2c-e5dc-4421-aaf1-c3e561e1bf0a service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:37:15.206759 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43392906-cd09-4136-80cc-12f5814bb4bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:37:15.208537 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43392906-cd09-4136-80cc-12f5814bb4bf service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:37:15.208537 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43392906-cd09-4136-80cc-12f5814bb4bf service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:37:15.208537 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43392906-cd09-4136-80cc-12f5814bb4bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:37:15.209751 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-43392906-cd09-4136-80cc-12f5814bb4bf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:37:15.220882 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43392906-cd09-4136-80cc-12f5814bb4bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:37:15.222346 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43392906-cd09-4136-80cc-12f5814bb4bf service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:37:15.239490 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1404/2825] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:37:15 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:37:15.788772 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7858630b-81c4-4f88-8269-4c21c7ed8be9 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:37:15.809990 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7858630b-81c4-4f88-8269-4c21c7ed8be9 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:37:15.818290 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6e4710-169c-4cba-9238-cd6f29d76597 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:37:15.819280 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6e4710-169c-4cba-9238-cd6f29d76597 service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:37:15.819763 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6e4710-169c-4cba-9238-cd6f29d76597 service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:37:15.820253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6e4710-169c-4cba-9238-cd6f29d76597 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:37:15.823855 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-ff6e4710-169c-4cba-9238-cd6f29d76597 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:37:15.845476 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6e4710-169c-4cba-9238-cd6f29d76597 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:37:15.847178 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6e4710-169c-4cba-9238-cd6f29d76597 service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:37:15.858456 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1422/2826] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:37:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:37:23.434492 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43392906-cd09-4136-80cc-12f5814bb4bf service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:37:23.435984 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-43392906-cd09-4136-80cc-12f5814bb4bf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:37:23.452369 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43392906-cd09-4136-80cc-12f5814bb4bf service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:37:23.457888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801790ee-5b5d-459e-bdfb-72a3183edb4e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:37:23.458488 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801790ee-5b5d-459e-bdfb-72a3183edb4e service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:37:23.458770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801790ee-5b5d-459e-bdfb-72a3183edb4e service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:37:23.459042 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801790ee-5b5d-459e-bdfb-72a3183edb4e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:37:23.460753 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-801790ee-5b5d-459e-bdfb-72a3183edb4e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:37:23.473972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801790ee-5b5d-459e-bdfb-72a3183edb4e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:37:23.475555 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801790ee-5b5d-459e-bdfb-72a3183edb4e service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:37:23.486763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1405/2827] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:37:23 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:37:25.853715 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=9 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:37:32.461876 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6e4710-169c-4cba-9238-cd6f29d76597 service nova] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:37:32.472036 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6e4710-169c-4cba-9238-cd6f29d76597 service nova] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:37:32.475696 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c34169-03df-4610-8145-1021fa7562d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:37:32.476288 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c34169-03df-4610-8145-1021fa7562d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:37:32.476567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c34169-03df-4610-8145-1021fa7562d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:37:32.476835 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c34169-03df-4610-8145-1021fa7562d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:37:32.479125 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-81c34169-03df-4610-8145-1021fa7562d2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:37:32.490989 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c34169-03df-4610-8145-1021fa7562d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:37:32.492411 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c34169-03df-4610-8145-1021fa7562d2 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:37:32.502555 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1423/2828] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:37:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:37:33.482734 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=12 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:37:35.301141 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801790ee-5b5d-459e-bdfb-72a3183edb4e service nova] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:37:35.303458 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-801790ee-5b5d-459e-bdfb-72a3183edb4e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:37:35.396858 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801790ee-5b5d-459e-bdfb-72a3183edb4e service nova] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:37:35.402629 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6169bf25-3733-4c78-9528-200982cbaf9e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '359d79cbfcad49beba2b495066824128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:37:35.403231 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6169bf25-3733-4c78-9528-200982cbaf9e service placement] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:37:35.403509 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6169bf25-3733-4c78-9528-200982cbaf9e service placement] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:37:35.403776 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6169bf25-3733-4c78-9528-200982cbaf9e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:37:35.405523 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6169bf25-3733-4c78-9528-200982cbaf9e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:37:35.419247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6169bf25-3733-4c78-9528-200982cbaf9e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:37:35.420851 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6169bf25-3733-4c78-9528-200982cbaf9e service placement] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:37:35.430938 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1406/2829] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:37:35 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:37:38.680184 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c34169-03df-4610-8145-1021fa7562d2 service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:37:38.692057 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c34169-03df-4610-8145-1021fa7562d2 service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:37:38.697164 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b423c5-0738-4056-ac86-f8f41bfdaa14 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '359d79cbfcad49beba2b495066824128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:37:38.697974 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b423c5-0738-4056-ac86-f8f41bfdaa14 service placement] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:37:38.698247 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b423c5-0738-4056-ac86-f8f41bfdaa14 service placement] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:37:38.698510 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b423c5-0738-4056-ac86-f8f41bfdaa14 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:37:38.700231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-85b423c5-0738-4056-ac86-f8f41bfdaa14 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:37:38.711721 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b423c5-0738-4056-ac86-f8f41bfdaa14 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:37:38.713344 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b423c5-0738-4056-ac86-f8f41bfdaa14 service placement] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:37:38.723237 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1424/2830] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:37:38 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:37:38.742346 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6169bf25-3733-4c78-9528-200982cbaf9e service placement] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:37:38.744474 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6169bf25-3733-4c78-9528-200982cbaf9e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:37:38.759591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6169bf25-3733-4c78-9528-200982cbaf9e service placement] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:37:38.769574 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2891351-4ce8-4e22-af5b-42919815fb07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:37:38.770062 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2891351-4ce8-4e22-af5b-42919815fb07 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:37:38.770342 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2891351-4ce8-4e22-af5b-42919815fb07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:37:38.770633 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2891351-4ce8-4e22-af5b-42919815fb07 admin admin] PATH_INFO: `/v3/users/a71c0e86c4b2456abce0e459705a2919` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:37:38.771878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2891351-4ce8-4e22-af5b-42919815fb07 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a71c0e86c4b2456abce0e459705a2919)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:37:38.774009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2891351-4ce8-4e22-af5b-42919815fb07 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:37:38.821385 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b2891351-4ce8-4e22-af5b-42919815fb07 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a71c0e86c4b2456abce0e459705a2919'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:38.833424 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b2891351-4ce8-4e22-af5b-42919815fb07 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a71c0e86c4b2456abce0e459705a2919'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:38.840830 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b2891351-4ce8-4e22-af5b-42919815fb07 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a71c0e86c4b2456abce0e459705a2919'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:38.848301 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b2891351-4ce8-4e22-af5b-42919815fb07 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a71c0e86c4b2456abce0e459705a2919'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:38.848922 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b2891351-4ce8-4e22-af5b-42919815fb07 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a71c0e86c4b2456abce0e459705a2919'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:38.855448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1407/2831] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:37:38 2022] DELETE /identity/v3/users/a71c0e86c4b2456abce0e459705a2919 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:37:39.028698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b423c5-0738-4056-ac86-f8f41bfdaa14 service placement] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:37:39.040321 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b423c5-0738-4056-ac86-f8f41bfdaa14 service placement] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:37:39.097348 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03a03cc-ef6f-4afa-bc38-e78cd2b4230f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:37:39.097975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03a03cc-ef6f-4afa-bc38-e78cd2b4230f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:37:39.098252 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03a03cc-ef6f-4afa-bc38-e78cd2b4230f admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:37:39.098522 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03a03cc-ef6f-4afa-bc38-e78cd2b4230f admin admin] PATH_INFO: `/v3/projects/96907629895d451697f448cf2c3bc6a1` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:37:39.099617 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03a03cc-ef6f-4afa-bc38-e78cd2b4230f admin admin] RBAC: Authorizing `identity:delete_project(project_id=96907629895d451697f448cf2c3bc6a1)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:37:39.101548 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03a03cc-ef6f-4afa-bc38-e78cd2b4230f admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:37:39.147743 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a03a03cc-ef6f-4afa-bc38-e78cd2b4230f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '96907629895d451697f448cf2c3bc6a1', 'request_id': 'req-a03a03cc-ef6f-4afa-bc38-e78cd2b4230f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:39.153243 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a03a03cc-ef6f-4afa-bc38-e78cd2b4230f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '96907629895d451697f448cf2c3bc6a1', 'request_id': 'req-a03a03cc-ef6f-4afa-bc38-e78cd2b4230f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:39.160705 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a03a03cc-ef6f-4afa-bc38-e78cd2b4230f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '96907629895d451697f448cf2c3bc6a1', 'request_id': 'req-a03a03cc-ef6f-4afa-bc38-e78cd2b4230f'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:39.172662 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a03a03cc-ef6f-4afa-bc38-e78cd2b4230f admin admin] The token cache is being invalidate because project 96907629895d451697f448cf2c3bc6a1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:37:39.172953 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a03a03cc-ef6f-4afa-bc38-e78cd2b4230f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:39.173605 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-a03a03cc-ef6f-4afa-bc38-e78cd2b4230f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:39.175079 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1425/2832] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:37:39 2022] DELETE /identity/v3/projects/96907629895d451697f448cf2c3bc6a1 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:37:39.179207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2891351-4ce8-4e22-af5b-42919815fb07 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:37:39.180597 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b2891351-4ce8-4e22-af5b-42919815fb07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:37:39.194378 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2891351-4ce8-4e22-af5b-42919815fb07 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:37:39.251025 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc7b6ab-a2b2-4a9f-8b71-dd8b29894fe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:37:39.251650 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc7b6ab-a2b2-4a9f-8b71-dd8b29894fe3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:37:39.251932 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc7b6ab-a2b2-4a9f-8b71-dd8b29894fe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:37:39.252210 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc7b6ab-a2b2-4a9f-8b71-dd8b29894fe3 admin admin] PATH_INFO: `/v3/users/1f965e6aca9341edbe6ecd391666993d` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:37:39.253370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc7b6ab-a2b2-4a9f-8b71-dd8b29894fe3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1f965e6aca9341edbe6ecd391666993d)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:37:39.255288 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc7b6ab-a2b2-4a9f-8b71-dd8b29894fe3 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:37:39.257868 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:37:39.303472 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8dc7b6ab-a2b2-4a9f-8b71-dd8b29894fe3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1f965e6aca9341edbe6ecd391666993d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:39.316342 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8dc7b6ab-a2b2-4a9f-8b71-dd8b29894fe3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1f965e6aca9341edbe6ecd391666993d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:39.324401 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8dc7b6ab-a2b2-4a9f-8b71-dd8b29894fe3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1f965e6aca9341edbe6ecd391666993d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:39.331804 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8dc7b6ab-a2b2-4a9f-8b71-dd8b29894fe3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1f965e6aca9341edbe6ecd391666993d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:39.332568 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8dc7b6ab-a2b2-4a9f-8b71-dd8b29894fe3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1f965e6aca9341edbe6ecd391666993d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:39.338495 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1408/2833] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:37:39 2022] DELETE /identity/v3/users/1f965e6aca9341edbe6ecd391666993d => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:37:39.507443 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03a03cc-ef6f-4afa-bc38-e78cd2b4230f admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:37:39.509048 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-a03a03cc-ef6f-4afa-bc38-e78cd2b4230f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:37:39.525208 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03a03cc-ef6f-4afa-bc38-e78cd2b4230f admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:37:39.581938 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8de6a45-2cc5-4f5f-bafc-a71fbe1f9c40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:37:39.582494 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8de6a45-2cc5-4f5f-bafc-a71fbe1f9c40 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:37:39.582771 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8de6a45-2cc5-4f5f-bafc-a71fbe1f9c40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:37:39.583042 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8de6a45-2cc5-4f5f-bafc-a71fbe1f9c40 admin admin] PATH_INFO: `/v3/projects/1b05404af98c48dd9c26aafd234846a9` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:37:39.584143 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8de6a45-2cc5-4f5f-bafc-a71fbe1f9c40 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1b05404af98c48dd9c26aafd234846a9)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:37:39.586081 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8de6a45-2cc5-4f5f-bafc-a71fbe1f9c40 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:37:39.592744 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:37:39.634127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c8de6a45-2cc5-4f5f-bafc-a71fbe1f9c40 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1b05404af98c48dd9c26aafd234846a9', 'request_id': 'req-c8de6a45-2cc5-4f5f-bafc-a71fbe1f9c40'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:39.639637 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c8de6a45-2cc5-4f5f-bafc-a71fbe1f9c40 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1b05404af98c48dd9c26aafd234846a9', 'request_id': 'req-c8de6a45-2cc5-4f5f-bafc-a71fbe1f9c40'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:39.646886 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c8de6a45-2cc5-4f5f-bafc-a71fbe1f9c40 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1b05404af98c48dd9c26aafd234846a9', 'request_id': 'req-c8de6a45-2cc5-4f5f-bafc-a71fbe1f9c40'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:39.658370 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c8de6a45-2cc5-4f5f-bafc-a71fbe1f9c40 admin admin] The token cache is being invalidate because project 1b05404af98c48dd9c26aafd234846a9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:37:39.658652 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c8de6a45-2cc5-4f5f-bafc-a71fbe1f9c40 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:39.659298 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c8de6a45-2cc5-4f5f-bafc-a71fbe1f9c40 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:37:39.660776 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1426/2834] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:37:39 2022] DELETE /identity/v3/projects/1b05404af98c48dd9c26aafd234846a9 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:37:49.335004 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=25,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:37:49.655282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=22,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:37:52.279709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c87b62-08f2-4e00-8d42-f6ca953392df None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:37:52.280637 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c87b62-08f2-4e00-8d42-f6ca953392df None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:37:52.281193 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c87b62-08f2-4e00-8d42-f6ca953392df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:37:52.301668 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-88c87b62-08f2-4e00-8d42-f6ca953392df None None] MFA Rules not processed for user `2625939a0b9a43d59452df9ba9982ecb`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:37:52.302878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-88c87b62-08f2-4e00-8d42-f6ca953392df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:37:52.356758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1409/2835] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:37:52 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:37:52.666765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8de6a45-2cc5-4f5f-bafc-a71fbe1f9c40 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:37:52.668289 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c8de6a45-2cc5-4f5f-bafc-a71fbe1f9c40 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:37:52.723873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8de6a45-2cc5-4f5f-bafc-a71fbe1f9c40 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:37:52.727408 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12053b66-34ee-41b7-a497-72c8fbc2c0f9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:37:52.727994 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12053b66-34ee-41b7-a497-72c8fbc2c0f9 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:37:52.728268 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12053b66-34ee-41b7-a497-72c8fbc2c0f9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:37:52.728537 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12053b66-34ee-41b7-a497-72c8fbc2c0f9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:37:52.730717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-12053b66-34ee-41b7-a497-72c8fbc2c0f9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:37:52.740037 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12053b66-34ee-41b7-a497-72c8fbc2c0f9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:37:52.741483 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12053b66-34ee-41b7-a497-72c8fbc2c0f9 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:37:52.750004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1427/2836] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:37:52 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:37:53.203143 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c87b62-08f2-4e00-8d42-f6ca953392df None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:37:53.204669 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-88c87b62-08f2-4e00-8d42-f6ca953392df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:37:53.276934 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c87b62-08f2-4e00-8d42-f6ca953392df None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:37:53.281973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e355c32-bcbb-4b08-914b-8a82979a1b73 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '359d79cbfcad49beba2b495066824128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:37:53.282579 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e355c32-bcbb-4b08-914b-8a82979a1b73 service placement] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:37:53.282859 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e355c32-bcbb-4b08-914b-8a82979a1b73 service placement] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:37:53.283129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e355c32-bcbb-4b08-914b-8a82979a1b73 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:37:53.285012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1e355c32-bcbb-4b08-914b-8a82979a1b73 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:37:53.297233 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e355c32-bcbb-4b08-914b-8a82979a1b73 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:37:53.298800 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e355c32-bcbb-4b08-914b-8a82979a1b73 service placement] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:37:53.308414 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1410/2837] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:37:53 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:38:02.746547 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=26 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:38:03.304274 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=57 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:38:06.329961 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12053b66-34ee-41b7-a497-72c8fbc2c0f9 service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:38:06.331391 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-12053b66-34ee-41b7-a497-72c8fbc2c0f9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:38:06.411745 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12053b66-34ee-41b7-a497-72c8fbc2c0f9 service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:38:06.417184 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592d9928-21ad-4f2c-a6ee-c2415d9037f4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:38:06.418527 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592d9928-21ad-4f2c-a6ee-c2415d9037f4 service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:38:06.419003 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592d9928-21ad-4f2c-a6ee-c2415d9037f4 service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:38:06.419410 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592d9928-21ad-4f2c-a6ee-c2415d9037f4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:38:06.421215 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-592d9928-21ad-4f2c-a6ee-c2415d9037f4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:38:06.433105 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592d9928-21ad-4f2c-a6ee-c2415d9037f4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:38:06.435273 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592d9928-21ad-4f2c-a6ee-c2415d9037f4 service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:38:06.445573 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1428/2838] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:38:06 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:38:16.441240 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=34 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:38:22.238220 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c3a165-b047-4cf7-b2d3-f987b175a06c None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:38:22.238726 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c3a165-b047-4cf7-b2d3-f987b175a06c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:38:22.238867 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c3a165-b047-4cf7-b2d3-f987b175a06c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:38:22.258563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-11c3a165-b047-4cf7-b2d3-f987b175a06c None None] MFA Rules not processed for user `8df68cb66f6d4369bba0eda189b8e64d`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:38:22.259788 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-11c3a165-b047-4cf7-b2d3-f987b175a06c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:38:22.313456 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1411/2839] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:38:22 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:38:32.306942 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=25 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:39:27.250824 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592d9928-21ad-4f2c-a6ee-c2415d9037f4 service nova] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:39:27.252232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-592d9928-21ad-4f2c-a6ee-c2415d9037f4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:39:27.266175 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592d9928-21ad-4f2c-a6ee-c2415d9037f4 service nova] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:39:27.271901 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4100863-9359-4643-abf3-c193a845bf94 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '359d79cbfcad49beba2b495066824128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:39:27.272502 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4100863-9359-4643-abf3-c193a845bf94 service placement] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:39:27.272790 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4100863-9359-4643-abf3-c193a845bf94 service placement] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:39:27.273082 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4100863-9359-4643-abf3-c193a845bf94 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:39:27.274874 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f4100863-9359-4643-abf3-c193a845bf94 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:39:27.289553 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4100863-9359-4643-abf3-c193a845bf94 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:39:27.291165 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4100863-9359-4643-abf3-c193a845bf94 service placement] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:39:27.302418 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1429/2840] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:39:27 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:39:30.910763 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd51503b-2ef0-4e62-8ba2-c6fe5d85fb79 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:39:30.911351 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd51503b-2ef0-4e62-8ba2-c6fe5d85fb79 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:39:30.911351 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd51503b-2ef0-4e62-8ba2-c6fe5d85fb79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:39:30.930337 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-dd51503b-2ef0-4e62-8ba2-c6fe5d85fb79 None None] MFA Rules not processed for user `8df68cb66f6d4369bba0eda189b8e64d`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:39:30.931554 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-dd51503b-2ef0-4e62-8ba2-c6fe5d85fb79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:39:30.938415 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1412/2841] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:39:30 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:39:37.298743 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=6 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:39:40.924823 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:39:58.995963 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4100863-9359-4643-abf3-c193a845bf94 service placement] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:39:58.997442 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f4100863-9359-4643-abf3-c193a845bf94 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:39:59.054491 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4100863-9359-4643-abf3-c193a845bf94 service placement] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:39:59.058127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738cf777-5da9-4e75-9b8c-bf7e7f11a66e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:39:59.058733 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738cf777-5da9-4e75-9b8c-bf7e7f11a66e service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:39:59.059013 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738cf777-5da9-4e75-9b8c-bf7e7f11a66e service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:39:59.059285 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738cf777-5da9-4e75-9b8c-bf7e7f11a66e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:39:59.061062 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-738cf777-5da9-4e75-9b8c-bf7e7f11a66e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:39:59.071906 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738cf777-5da9-4e75-9b8c-bf7e7f11a66e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:39:59.073366 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738cf777-5da9-4e75-9b8c-bf7e7f11a66e service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:39:59.082571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1430/2842] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:39:58 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:40:05.818507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d2c8b6-b31f-4894-8dd2-56e6027b61f7 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:05.819532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d2c8b6-b31f-4894-8dd2-56e6027b61f7 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:05.819891 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d2c8b6-b31f-4894-8dd2-56e6027b61f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:05.839192 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-31d2c8b6-b31f-4894-8dd2-56e6027b61f7 None None] MFA Rules not processed for user `2625939a0b9a43d59452df9ba9982ecb`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:40:05.840398 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-31d2c8b6-b31f-4894-8dd2-56e6027b61f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:40:05.847218 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1413/2843] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:40:05 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:40:06.181954 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738cf777-5da9-4e75-9b8c-bf7e7f11a66e service cinder] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:40:06.191556 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738cf777-5da9-4e75-9b8c-bf7e7f11a66e service cinder] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:40:06.195123 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81229beb-2da6-41d1-a19e-1ea20d829a36 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:40:06.195719 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81229beb-2da6-41d1-a19e-1ea20d829a36 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:06.196001 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81229beb-2da6-41d1-a19e-1ea20d829a36 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:06.196271 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81229beb-2da6-41d1-a19e-1ea20d829a36 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:06.197990 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-81229beb-2da6-41d1-a19e-1ea20d829a36 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:40:06.208164 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81229beb-2da6-41d1-a19e-1ea20d829a36 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:40:06.209657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81229beb-2da6-41d1-a19e-1ea20d829a36 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:40:06.217754 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1431/2844] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:40:06 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:40:15.833920 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:40:16.214058 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=32 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:40:51.425618 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d2c8b6-b31f-4894-8dd2-56e6027b61f7 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:40:51.427591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-31d2c8b6-b31f-4894-8dd2-56e6027b61f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:40:51.500437 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d2c8b6-b31f-4894-8dd2-56e6027b61f7 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:40:51.505691 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be47c8d-9e97-4c33-ac5d-786cf60015af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:40:51.506323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be47c8d-9e97-4c33-ac5d-786cf60015af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:51.506597 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be47c8d-9e97-4c33-ac5d-786cf60015af admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:51.506864 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be47c8d-9e97-4c33-ac5d-786cf60015af admin admin] PATH_INFO: `/v3/users/8df68cb66f6d4369bba0eda189b8e64d` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:51.507978 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be47c8d-9e97-4c33-ac5d-786cf60015af admin admin] RBAC: Authorizing `identity:delete_user(user_id=8df68cb66f6d4369bba0eda189b8e64d)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:40:51.509961 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be47c8d-9e97-4c33-ac5d-786cf60015af admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:40:51.560493 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2be47c8d-9e97-4c33-ac5d-786cf60015af admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8df68cb66f6d4369bba0eda189b8e64d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:40:51.573226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2be47c8d-9e97-4c33-ac5d-786cf60015af admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8df68cb66f6d4369bba0eda189b8e64d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:40:51.580548 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2be47c8d-9e97-4c33-ac5d-786cf60015af admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8df68cb66f6d4369bba0eda189b8e64d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:40:51.588827 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2be47c8d-9e97-4c33-ac5d-786cf60015af admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8df68cb66f6d4369bba0eda189b8e64d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:40:51.589473 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2be47c8d-9e97-4c33-ac5d-786cf60015af admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8df68cb66f6d4369bba0eda189b8e64d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:40:51.595856 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1414/2845] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:40:51 2022] DELETE /identity/v3/users/8df68cb66f6d4369bba0eda189b8e64d => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:40:51.772292 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81229beb-2da6-41d1-a19e-1ea20d829a36 service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:40:51.787524 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81229beb-2da6-41d1-a19e-1ea20d829a36 service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:40:51.850100 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58e1570-3911-47e5-938c-02fc3e028fcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:40:51.850690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58e1570-3911-47e5-938c-02fc3e028fcd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:51.850973 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58e1570-3911-47e5-938c-02fc3e028fcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:51.851255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58e1570-3911-47e5-938c-02fc3e028fcd admin admin] PATH_INFO: `/v3/projects/c561fdf35e454833b521f54043c49035` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:51.853523 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58e1570-3911-47e5-938c-02fc3e028fcd admin admin] RBAC: Authorizing `identity:delete_project(project_id=c561fdf35e454833b521f54043c49035)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:40:51.854410 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58e1570-3911-47e5-938c-02fc3e028fcd admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:40:51.905967 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e58e1570-3911-47e5-938c-02fc3e028fcd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c561fdf35e454833b521f54043c49035', 'request_id': 'req-e58e1570-3911-47e5-938c-02fc3e028fcd'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:40:51.912459 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e58e1570-3911-47e5-938c-02fc3e028fcd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c561fdf35e454833b521f54043c49035', 'request_id': 'req-e58e1570-3911-47e5-938c-02fc3e028fcd'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:40:51.921584 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e58e1570-3911-47e5-938c-02fc3e028fcd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c561fdf35e454833b521f54043c49035', 'request_id': 'req-e58e1570-3911-47e5-938c-02fc3e028fcd'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:40:51.933818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e58e1570-3911-47e5-938c-02fc3e028fcd admin admin] The token cache is being invalidate because project c561fdf35e454833b521f54043c49035 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:40:51.934197 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e58e1570-3911-47e5-938c-02fc3e028fcd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:40:51.934847 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e58e1570-3911-47e5-938c-02fc3e028fcd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:40:51.936334 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1432/2846] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:40:51 2022] DELETE /identity/v3/projects/c561fdf35e454833b521f54043c49035 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:40:51.941845 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d4e81e-7bc5-4e12-a3e4-44300d6d5464 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:51.942129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d4e81e-7bc5-4e12-a3e4-44300d6d5464 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:51.942400 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d4e81e-7bc5-4e12-a3e4-44300d6d5464 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:51.964137 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-37d4e81e-7bc5-4e12-a3e4-44300d6d5464 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:40:51.965421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-37d4e81e-7bc5-4e12-a3e4-44300d6d5464 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:40:52.026744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1415/2847] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:40:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:40:52.035002 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b660acb0-329e-4fc3-91ca-f88afb489f3d None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:52.035280 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b660acb0-329e-4fc3-91ca-f88afb489f3d None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:52.035577 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b660acb0-329e-4fc3-91ca-f88afb489f3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:52.056300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-b660acb0-329e-4fc3-91ca-f88afb489f3d None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:40:52.057542 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b660acb0-329e-4fc3-91ca-f88afb489f3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:40:52.065438 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1433/2848] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:40:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:40:52.070685 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d4e81e-7bc5-4e12-a3e4-44300d6d5464 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:40:52.080572 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d4e81e-7bc5-4e12-a3e4-44300d6d5464 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:40:52.086073 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c278e922-65b1-4c8a-b9ad-6fbb8dde3f71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:40:52.086683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c278e922-65b1-4c8a-b9ad-6fbb8dde3f71 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:52.086966 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c278e922-65b1-4c8a-b9ad-6fbb8dde3f71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:52.087246 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c278e922-65b1-4c8a-b9ad-6fbb8dde3f71 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:52.087902 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c278e922-65b1-4c8a-b9ad-6fbb8dde3f71 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:40:52.088206 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c278e922-65b1-4c8a-b9ad-6fbb8dde3f71 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:40:52.089865 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c278e922-65b1-4c8a-b9ad-6fbb8dde3f71 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:40:52.102144 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1416/2849] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:40:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:40:52.106529 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b660acb0-329e-4fc3-91ca-f88afb489f3d None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:40:52.117586 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b660acb0-329e-4fc3-91ca-f88afb489f3d None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:40:52.122453 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84d2ce8-787f-45a2-8177-215f6f900f8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:40:52.123093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84d2ce8-787f-45a2-8177-215f6f900f8d admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:52.123370 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84d2ce8-787f-45a2-8177-215f6f900f8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:52.123657 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84d2ce8-787f-45a2-8177-215f6f900f8d admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:52.124327 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84d2ce8-787f-45a2-8177-215f6f900f8d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:40:52.126294 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84d2ce8-787f-45a2-8177-215f6f900f8d admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:40:52.168993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1434/2850] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:40:52 2022] POST /identity/v3/projects => generated 371 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:40:52.170689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c278e922-65b1-4c8a-b9ad-6fbb8dde3f71 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:40:52.181478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c278e922-65b1-4c8a-b9ad-6fbb8dde3f71 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:40:52.185577 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:40:52.242070 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bf3bb1-8446-4570-944b-05d72110d4f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:40:52.242684 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bf3bb1-8446-4570-944b-05d72110d4f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:52.242964 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bf3bb1-8446-4570-944b-05d72110d4f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:52.243243 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bf3bb1-8446-4570-944b-05d72110d4f9 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:52.243887 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01bf3bb1-8446-4570-944b-05d72110d4f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:40:52.245844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01bf3bb1-8446-4570-944b-05d72110d4f9 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:40:52.271337 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1417/2851] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:40:52 2022] POST /identity/v3/users => generated 286 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:40:52.275596 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84d2ce8-787f-45a2-8177-215f6f900f8d admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:40:52.287111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84d2ce8-787f-45a2-8177-215f6f900f8d admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:40:52.293568 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546eee1b-0fd6-4f96-aec9-b32574ed3eb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:40:52.294178 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546eee1b-0fd6-4f96-aec9-b32574ed3eb5 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:52.294460 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546eee1b-0fd6-4f96-aec9-b32574ed3eb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:52.294739 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546eee1b-0fd6-4f96-aec9-b32574ed3eb5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:52.295382 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546eee1b-0fd6-4f96-aec9-b32574ed3eb5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:40:52.297014 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546eee1b-0fd6-4f96-aec9-b32574ed3eb5 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:40:52.307993 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1435/2852] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:40:52 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:40:52.312338 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bf3bb1-8446-4570-944b-05d72110d4f9 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:40:52.322427 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bf3bb1-8446-4570-944b-05d72110d4f9 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:40:52.328811 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862918b6-ebe2-4f30-be25-100c893a899f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:40:52.329426 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862918b6-ebe2-4f30-be25-100c893a899f admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:52.329707 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862918b6-ebe2-4f30-be25-100c893a899f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:52.329980 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862918b6-ebe2-4f30-be25-100c893a899f admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:52.330629 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862918b6-ebe2-4f30-be25-100c893a899f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:40:52.332769 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862918b6-ebe2-4f30-be25-100c893a899f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:40:52.343934 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1418/2853] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:40:52 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:40:52.348923 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546eee1b-0fd6-4f96-aec9-b32574ed3eb5 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:40:52.362216 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546eee1b-0fd6-4f96-aec9-b32574ed3eb5 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:40:52.367027 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ea38e6-af54-4b27-8855-f219858c5c27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:40:52.367695 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ea38e6-af54-4b27-8855-f219858c5c27 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:52.367981 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ea38e6-af54-4b27-8855-f219858c5c27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:52.368258 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ea38e6-af54-4b27-8855-f219858c5c27 admin admin] PATH_INFO: `/v3/projects/456fbce182e94b168726fbf36bab00c8/users/2d10facf9759450a8229d8bd17b3ac03/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:52.384631 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ea38e6-af54-4b27-8855-f219858c5c27 admin admin] RBAC: Authorizing `identity:create_grant(project_id=456fbce182e94b168726fbf36bab00c8, user_id=2d10facf9759450a8229d8bd17b3ac03, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:40:52.387335 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ea38e6-af54-4b27-8855-f219858c5c27 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:40:52.400956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1436/2854] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:40:52 2022] PUT /identity/v3/projects/456fbce182e94b168726fbf36bab00c8/users/2d10facf9759450a8229d8bd17b3ac03/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:40:52.412075 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae132102-61d8-44bc-a231-42e5d1e7f097 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:52.412359 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae132102-61d8-44bc-a231-42e5d1e7f097 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:52.412638 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae132102-61d8-44bc-a231-42e5d1e7f097 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:52.441131 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ae132102-61d8-44bc-a231-42e5d1e7f097 None None] MFA Rules not processed for user `2d10facf9759450a8229d8bd17b3ac03`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:40:52.442907 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ae132102-61d8-44bc-a231-42e5d1e7f097 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:40:52.509710 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1419/2855] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:40:52 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:40:52.514264 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ea38e6-af54-4b27-8855-f219858c5c27 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:40:52.534141 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:40:52.584908 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ea38e6-af54-4b27-8855-f219858c5c27 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:40:52.590250 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13168e7-5b3a-45f7-94a0-455eb3d95993 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:40:52.590892 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13168e7-5b3a-45f7-94a0-455eb3d95993 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:52.591178 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13168e7-5b3a-45f7-94a0-455eb3d95993 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:52.592426 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13168e7-5b3a-45f7-94a0-455eb3d95993 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:52.593107 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13168e7-5b3a-45f7-94a0-455eb3d95993 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:40:52.595044 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13168e7-5b3a-45f7-94a0-455eb3d95993 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:40:52.624348 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1437/2856] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:40:52 2022] POST /identity/v3/projects => generated 373 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:40:52.628304 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae132102-61d8-44bc-a231-42e5d1e7f097 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:40:52.643539 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae132102-61d8-44bc-a231-42e5d1e7f097 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:40:52.703661 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57ce4f5-5167-4c24-b53a-b8bd5462043b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:40:52.704270 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57ce4f5-5167-4c24-b53a-b8bd5462043b admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:52.704556 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57ce4f5-5167-4c24-b53a-b8bd5462043b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:52.704844 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57ce4f5-5167-4c24-b53a-b8bd5462043b admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:52.705526 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57ce4f5-5167-4c24-b53a-b8bd5462043b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:40:52.707630 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57ce4f5-5167-4c24-b53a-b8bd5462043b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:40:52.730822 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1420/2857] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:40:52 2022] POST /identity/v3/users => generated 293 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:40:52.735220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13168e7-5b3a-45f7-94a0-455eb3d95993 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:40:52.746502 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13168e7-5b3a-45f7-94a0-455eb3d95993 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:40:52.751213 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f34c03-abf2-411e-b5a8-cc39f3b7ad2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:40:52.751821 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f34c03-abf2-411e-b5a8-cc39f3b7ad2f admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:52.752101 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f34c03-abf2-411e-b5a8-cc39f3b7ad2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:52.752375 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f34c03-abf2-411e-b5a8-cc39f3b7ad2f admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:52.753019 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f34c03-abf2-411e-b5a8-cc39f3b7ad2f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:40:52.754658 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f34c03-abf2-411e-b5a8-cc39f3b7ad2f admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:40:52.765607 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1438/2858] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:40:52 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:40:52.770052 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57ce4f5-5167-4c24-b53a-b8bd5462043b admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:40:52.776515 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:40:52.780665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57ce4f5-5167-4c24-b53a-b8bd5462043b admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:40:52.787493 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809eaed2-bfa7-4554-a827-69e7548e3b35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:40:52.788160 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809eaed2-bfa7-4554-a827-69e7548e3b35 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:52.788442 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809eaed2-bfa7-4554-a827-69e7548e3b35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:52.788717 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809eaed2-bfa7-4554-a827-69e7548e3b35 admin admin] PATH_INFO: `/v3/projects/eb199a94bd7443a2b07676a40cace225/users/0be41fe4cdc8448ebd45644c3daa8702/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:52.804204 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809eaed2-bfa7-4554-a827-69e7548e3b35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb199a94bd7443a2b07676a40cace225, user_id=0be41fe4cdc8448ebd45644c3daa8702, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:40:52.806898 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809eaed2-bfa7-4554-a827-69e7548e3b35 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:40:52.820507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1421/2859] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:40:52 2022] PUT /identity/v3/projects/eb199a94bd7443a2b07676a40cace225/users/0be41fe4cdc8448ebd45644c3daa8702/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:40:52.825113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f34c03-abf2-411e-b5a8-cc39f3b7ad2f admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:40:52.835398 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f34c03-abf2-411e-b5a8-cc39f3b7ad2f admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:40:52.894215 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fe789c-a520-4e6d-b39a-ac097164c3cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:40:52.894810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fe789c-a520-4e6d-b39a-ac097164c3cf admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:52.895217 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fe789c-a520-4e6d-b39a-ac097164c3cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:52.895368 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fe789c-a520-4e6d-b39a-ac097164c3cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:52.896045 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fe789c-a520-4e6d-b39a-ac097164c3cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:40:52.897685 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fe789c-a520-4e6d-b39a-ac097164c3cf admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:40:52.907636 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1439/2860] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:40:52 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:40:52.912296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809eaed2-bfa7-4554-a827-69e7548e3b35 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:40:52.922557 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809eaed2-bfa7-4554-a827-69e7548e3b35 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:40:52.928594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73333299-f45b-40ab-beb8-86330d3908a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:40:52.929421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73333299-f45b-40ab-beb8-86330d3908a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:52.929703 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73333299-f45b-40ab-beb8-86330d3908a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:52.929972 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73333299-f45b-40ab-beb8-86330d3908a9 admin admin] PATH_INFO: `/v3/projects/eb199a94bd7443a2b07676a40cace225/users/0be41fe4cdc8448ebd45644c3daa8702/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:52.931886 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73333299-f45b-40ab-beb8-86330d3908a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb199a94bd7443a2b07676a40cace225, user_id=0be41fe4cdc8448ebd45644c3daa8702, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:40:52.934641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73333299-f45b-40ab-beb8-86330d3908a9 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:40:52.945680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1422/2861] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:40:52 2022] PUT /identity/v3/projects/eb199a94bd7443a2b07676a40cace225/users/0be41fe4cdc8448ebd45644c3daa8702/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:40:53.012282 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb7e800-a139-4f99-9360-8ab1b3f3f133 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:53.012568 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb7e800-a139-4f99-9360-8ab1b3f3f133 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:53.012846 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb7e800-a139-4f99-9360-8ab1b3f3f133 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:53.043179 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-aeb7e800-a139-4f99-9360-8ab1b3f3f133 None None] MFA Rules not processed for user `0be41fe4cdc8448ebd45644c3daa8702`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:40:53.044454 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-aeb7e800-a139-4f99-9360-8ab1b3f3f133 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:40:53.114601 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1440/2862] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:40:53 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:40:53.125832 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73333299-f45b-40ab-beb8-86330d3908a9 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:40:53.127564 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-73333299-f45b-40ab-beb8-86330d3908a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:40:53.187719 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73333299-f45b-40ab-beb8-86330d3908a9 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:40:53.191967 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d35c1e-4727-4067-a7ea-e6e2657bacb2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:40:53.192725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d35c1e-4727-4067-a7ea-e6e2657bacb2 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:53.193183 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d35c1e-4727-4067-a7ea-e6e2657bacb2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:53.193450 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d35c1e-4727-4067-a7ea-e6e2657bacb2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:53.203043 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d35c1e-4727-4067-a7ea-e6e2657bacb2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:40:53.204477 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d35c1e-4727-4067-a7ea-e6e2657bacb2 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:40:53.214683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1423/2863] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:40:53 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:40:53.296165 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb7e800-a139-4f99-9360-8ab1b3f3f133 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:40:53.308975 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb7e800-a139-4f99-9360-8ab1b3f3f133 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:40:53.312633 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ae6349-884a-44f4-a2ee-54873e1e6809 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:40:53.313267 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ae6349-884a-44f4-a2ee-54873e1e6809 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:40:53.313548 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ae6349-884a-44f4-a2ee-54873e1e6809 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:40:53.313821 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ae6349-884a-44f4-a2ee-54873e1e6809 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:40:53.328170 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:40:53.387113 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ae6349-884a-44f4-a2ee-54873e1e6809 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:40:53.388558 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ae6349-884a-44f4-a2ee-54873e1e6809 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:40:53.398429 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1441/2864] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:40:53 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:41:03.210294 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=39,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:41:03.394681 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=22 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:41:35.925989 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d35c1e-4727-4067-a7ea-e6e2657bacb2 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:35.987505 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d35c1e-4727-4067-a7ea-e6e2657bacb2 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:35.992299 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e4b3ce-51f6-4676-a2f8-d40b4186608f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:35.992924 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e4b3ce-51f6-4676-a2f8-d40b4186608f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:35.993247 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e4b3ce-51f6-4676-a2f8-d40b4186608f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:35.993518 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e4b3ce-51f6-4676-a2f8-d40b4186608f admin admin] PATH_INFO: `/v3/users/2d10facf9759450a8229d8bd17b3ac03` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:35.994628 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e4b3ce-51f6-4676-a2f8-d40b4186608f admin admin] RBAC: Authorizing `identity:delete_user(user_id=2d10facf9759450a8229d8bd17b3ac03)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:35.996540 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e4b3ce-51f6-4676-a2f8-d40b4186608f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:36.042652 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c2e4b3ce-51f6-4676-a2f8-d40b4186608f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2d10facf9759450a8229d8bd17b3ac03'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:36.054580 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c2e4b3ce-51f6-4676-a2f8-d40b4186608f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2d10facf9759450a8229d8bd17b3ac03'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:36.061282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c2e4b3ce-51f6-4676-a2f8-d40b4186608f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2d10facf9759450a8229d8bd17b3ac03'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:36.068469 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c2e4b3ce-51f6-4676-a2f8-d40b4186608f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2d10facf9759450a8229d8bd17b3ac03'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:36.069015 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c2e4b3ce-51f6-4676-a2f8-d40b4186608f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2d10facf9759450a8229d8bd17b3ac03'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:36.074902 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1424/2865] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:41:35 2022] DELETE /identity/v3/users/2d10facf9759450a8229d8bd17b3ac03 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:41:36.079402 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68cfc4f-cedd-446a-9b3b-d5e1bba164aa None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:36.079931 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68cfc4f-cedd-446a-9b3b-d5e1bba164aa None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:36.079931 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68cfc4f-cedd-446a-9b3b-d5e1bba164aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:36.100303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-a68cfc4f-cedd-446a-9b3b-d5e1bba164aa None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:41:36.101522 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-a68cfc4f-cedd-446a-9b3b-d5e1bba164aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:36.162028 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1442/2866] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:41:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:41:36.171720 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e4b3ce-51f6-4676-a2f8-d40b4186608f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:36.173183 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c2e4b3ce-51f6-4676-a2f8-d40b4186608f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:36.230159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e4b3ce-51f6-4676-a2f8-d40b4186608f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:36.234085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf63f8e1-a011-46b4-b2f3-8829c8cb2762 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:36.234688 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf63f8e1-a011-46b4-b2f3-8829c8cb2762 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:36.234967 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf63f8e1-a011-46b4-b2f3-8829c8cb2762 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:36.235239 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf63f8e1-a011-46b4-b2f3-8829c8cb2762 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:36.244216 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf63f8e1-a011-46b4-b2f3-8829c8cb2762 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:36.245723 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf63f8e1-a011-46b4-b2f3-8829c8cb2762 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:36.255432 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1425/2867] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:41:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:41:36.521317 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68cfc4f-cedd-446a-9b3b-d5e1bba164aa None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:36.523070 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-a68cfc4f-cedd-446a-9b3b-d5e1bba164aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:36.538212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68cfc4f-cedd-446a-9b3b-d5e1bba164aa None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:36.542983 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62b453d-accd-40fe-8b57-0cbca8c23bf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:36.543638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62b453d-accd-40fe-8b57-0cbca8c23bf8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:36.543888 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62b453d-accd-40fe-8b57-0cbca8c23bf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:36.544160 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62b453d-accd-40fe-8b57-0cbca8c23bf8 admin admin] PATH_INFO: `/v3/projects/456fbce182e94b168726fbf36bab00c8` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:36.545307 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62b453d-accd-40fe-8b57-0cbca8c23bf8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=456fbce182e94b168726fbf36bab00c8)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:36.547307 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62b453d-accd-40fe-8b57-0cbca8c23bf8 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:36.594956 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c62b453d-accd-40fe-8b57-0cbca8c23bf8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '456fbce182e94b168726fbf36bab00c8', 'request_id': 'req-c62b453d-accd-40fe-8b57-0cbca8c23bf8'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:36.600454 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c62b453d-accd-40fe-8b57-0cbca8c23bf8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '456fbce182e94b168726fbf36bab00c8', 'request_id': 'req-c62b453d-accd-40fe-8b57-0cbca8c23bf8'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:36.608225 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c62b453d-accd-40fe-8b57-0cbca8c23bf8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '456fbce182e94b168726fbf36bab00c8', 'request_id': 'req-c62b453d-accd-40fe-8b57-0cbca8c23bf8'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:36.619912 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c62b453d-accd-40fe-8b57-0cbca8c23bf8 admin admin] The token cache is being invalidate because project 456fbce182e94b168726fbf36bab00c8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:41:36.620198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c62b453d-accd-40fe-8b57-0cbca8c23bf8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:36.620862 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-c62b453d-accd-40fe-8b57-0cbca8c23bf8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:36.622432 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1443/2868] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:41:36 2022] DELETE /identity/v3/projects/456fbce182e94b168726fbf36bab00c8 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:41:36.626858 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf63f8e1-a011-46b4-b2f3-8829c8cb2762 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:36.637129 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf63f8e1-a011-46b4-b2f3-8829c8cb2762 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:36.654781 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:41:36.694821 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719ad585-7727-4f69-8ba8-837fd4bbe9ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:36.695449 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719ad585-7727-4f69-8ba8-837fd4bbe9ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:36.695770 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719ad585-7727-4f69-8ba8-837fd4bbe9ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:36.696043 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719ad585-7727-4f69-8ba8-837fd4bbe9ac admin admin] PATH_INFO: `/v3/users/0be41fe4cdc8448ebd45644c3daa8702` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:36.697230 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719ad585-7727-4f69-8ba8-837fd4bbe9ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=0be41fe4cdc8448ebd45644c3daa8702)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:36.699140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719ad585-7727-4f69-8ba8-837fd4bbe9ac admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:36.744561 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-719ad585-7727-4f69-8ba8-837fd4bbe9ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0be41fe4cdc8448ebd45644c3daa8702'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:36.758533 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-719ad585-7727-4f69-8ba8-837fd4bbe9ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0be41fe4cdc8448ebd45644c3daa8702'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:36.765662 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-719ad585-7727-4f69-8ba8-837fd4bbe9ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0be41fe4cdc8448ebd45644c3daa8702'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:36.773000 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-719ad585-7727-4f69-8ba8-837fd4bbe9ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0be41fe4cdc8448ebd45644c3daa8702'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:36.773673 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-719ad585-7727-4f69-8ba8-837fd4bbe9ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0be41fe4cdc8448ebd45644c3daa8702'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:36.779470 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1426/2869] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:41:36 2022] DELETE /identity/v3/users/0be41fe4cdc8448ebd45644c3daa8702 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:41:37.050380 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62b453d-accd-40fe-8b57-0cbca8c23bf8 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:37.052814 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c62b453d-accd-40fe-8b57-0cbca8c23bf8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:37.065902 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62b453d-accd-40fe-8b57-0cbca8c23bf8 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:37.127951 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fce08f-16f7-48de-99e9-440a22d43e10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:37.128652 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fce08f-16f7-48de-99e9-440a22d43e10 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:37.128873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fce08f-16f7-48de-99e9-440a22d43e10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:37.129235 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fce08f-16f7-48de-99e9-440a22d43e10 admin admin] PATH_INFO: `/v3/projects/eb199a94bd7443a2b07676a40cace225` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:37.130418 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fce08f-16f7-48de-99e9-440a22d43e10 admin admin] RBAC: Authorizing `identity:delete_project(project_id=eb199a94bd7443a2b07676a40cace225)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:37.132286 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fce08f-16f7-48de-99e9-440a22d43e10 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:37.163253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:41:37.181287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-27fce08f-16f7-48de-99e9-440a22d43e10 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eb199a94bd7443a2b07676a40cace225', 'request_id': 'req-27fce08f-16f7-48de-99e9-440a22d43e10'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:37.186748 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-27fce08f-16f7-48de-99e9-440a22d43e10 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eb199a94bd7443a2b07676a40cace225', 'request_id': 'req-27fce08f-16f7-48de-99e9-440a22d43e10'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:37.194724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-27fce08f-16f7-48de-99e9-440a22d43e10 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eb199a94bd7443a2b07676a40cace225', 'request_id': 'req-27fce08f-16f7-48de-99e9-440a22d43e10'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:37.205802 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-27fce08f-16f7-48de-99e9-440a22d43e10 admin admin] The token cache is being invalidate because project eb199a94bd7443a2b07676a40cace225 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:41:37.206091 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-27fce08f-16f7-48de-99e9-440a22d43e10 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:37.206720 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-27fce08f-16f7-48de-99e9-440a22d43e10 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:37.208196 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1444/2870] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:41:37 2022] DELETE /identity/v3/projects/eb199a94bd7443a2b07676a40cace225 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:41:37.213850 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6235d29-ad91-4b1e-b89a-82b5a9439a08 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:37.214369 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6235d29-ad91-4b1e-b89a-82b5a9439a08 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:37.214522 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6235d29-ad91-4b1e-b89a-82b5a9439a08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:37.234287 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-a6235d29-ad91-4b1e-b89a-82b5a9439a08 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:41:37.235509 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a6235d29-ad91-4b1e-b89a-82b5a9439a08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:37.293683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1427/2871] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:41:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:41:37.301567 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4154185-96d1-43c6-87d1-8ddb4a4db32f None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:37.301906 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4154185-96d1-43c6-87d1-8ddb4a4db32f None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:37.302688 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4154185-96d1-43c6-87d1-8ddb4a4db32f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:37.322787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-d4154185-96d1-43c6-87d1-8ddb4a4db32f None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:41:37.324059 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d4154185-96d1-43c6-87d1-8ddb4a4db32f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:37.332343 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1445/2872] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:41:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:41:37.336558 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6235d29-ad91-4b1e-b89a-82b5a9439a08 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:37.346232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6235d29-ad91-4b1e-b89a-82b5a9439a08 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:37.350817 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eece2a9c-22e7-4eca-a9cd-b53eed7f4864 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:37.351409 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eece2a9c-22e7-4eca-a9cd-b53eed7f4864 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:37.351683 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eece2a9c-22e7-4eca-a9cd-b53eed7f4864 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:37.351951 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eece2a9c-22e7-4eca-a9cd-b53eed7f4864 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:37.352595 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eece2a9c-22e7-4eca-a9cd-b53eed7f4864 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:41:37.352892 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eece2a9c-22e7-4eca-a9cd-b53eed7f4864 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:37.354535 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eece2a9c-22e7-4eca-a9cd-b53eed7f4864 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:37.366624 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1428/2873] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:41:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:41:37.370916 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4154185-96d1-43c6-87d1-8ddb4a4db32f None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:37.381141 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4154185-96d1-43c6-87d1-8ddb4a4db32f None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:37.385658 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb02945-79da-4c05-9395-14c29a38f1ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:37.386283 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb02945-79da-4c05-9395-14c29a38f1ab admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:37.386554 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb02945-79da-4c05-9395-14c29a38f1ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:37.386820 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb02945-79da-4c05-9395-14c29a38f1ab admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:37.387438 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb02945-79da-4c05-9395-14c29a38f1ab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:37.389419 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb02945-79da-4c05-9395-14c29a38f1ab admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:37.415873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1446/2874] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:41:37 2022] POST /identity/v3/projects => generated 365 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:41:37.420006 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eece2a9c-22e7-4eca-a9cd-b53eed7f4864 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:37.430594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eece2a9c-22e7-4eca-a9cd-b53eed7f4864 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:37.457313 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:41:37.490595 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce8ecb2-d273-49fd-9ce8-8d0216537e75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:37.491249 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce8ecb2-d273-49fd-9ce8-8d0216537e75 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:37.491532 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce8ecb2-d273-49fd-9ce8-8d0216537e75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:37.491801 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce8ecb2-d273-49fd-9ce8-8d0216537e75 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:37.492424 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce8ecb2-d273-49fd-9ce8-8d0216537e75 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:37.494414 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce8ecb2-d273-49fd-9ce8-8d0216537e75 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:37.516339 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1429/2875] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:41:37 2022] POST /identity/v3/users => generated 283 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:41:37.520708 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb02945-79da-4c05-9395-14c29a38f1ab admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:37.530995 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb02945-79da-4c05-9395-14c29a38f1ab admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:37.535695 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc17690d-c7eb-4234-8216-2c9a1c7f2a5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:37.536290 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc17690d-c7eb-4234-8216-2c9a1c7f2a5f admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:37.536569 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc17690d-c7eb-4234-8216-2c9a1c7f2a5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:37.536842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc17690d-c7eb-4234-8216-2c9a1c7f2a5f admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:37.537521 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc17690d-c7eb-4234-8216-2c9a1c7f2a5f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:37.539139 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc17690d-c7eb-4234-8216-2c9a1c7f2a5f admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:37.549024 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1447/2876] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:41:37 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:41:37.553640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce8ecb2-d273-49fd-9ce8-8d0216537e75 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:37.563790 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce8ecb2-d273-49fd-9ce8-8d0216537e75 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:37.569155 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f346b7d1-5ff4-4388-814c-e590d3bdda15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:37.569741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f346b7d1-5ff4-4388-814c-e590d3bdda15 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:37.570015 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f346b7d1-5ff4-4388-814c-e590d3bdda15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:37.570283 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f346b7d1-5ff4-4388-814c-e590d3bdda15 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:37.570914 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f346b7d1-5ff4-4388-814c-e590d3bdda15 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:37.572536 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f346b7d1-5ff4-4388-814c-e590d3bdda15 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:37.583235 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1430/2877] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:41:37 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:41:37.587957 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc17690d-c7eb-4234-8216-2c9a1c7f2a5f admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:37.597818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc17690d-c7eb-4234-8216-2c9a1c7f2a5f admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:37.603093 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3364457e-1c3e-428a-98f7-45854828c3ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:37.603734 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3364457e-1c3e-428a-98f7-45854828c3ca admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:37.604014 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3364457e-1c3e-428a-98f7-45854828c3ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:37.604295 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3364457e-1c3e-428a-98f7-45854828c3ca admin admin] PATH_INFO: `/v3/projects/0b64913b0b0f4521882b906a3c533c70/users/eb498be561b6468faac6bea1c12edb87/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:37.619585 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3364457e-1c3e-428a-98f7-45854828c3ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b64913b0b0f4521882b906a3c533c70, user_id=eb498be561b6468faac6bea1c12edb87, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:37.622309 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3364457e-1c3e-428a-98f7-45854828c3ca admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:37.636261 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1448/2878] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:41:37 2022] PUT /identity/v3/projects/0b64913b0b0f4521882b906a3c533c70/users/eb498be561b6468faac6bea1c12edb87/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:41:37.647160 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a1a1d4-c002-45d6-9573-fde4b07cb549 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:37.647452 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a1a1d4-c002-45d6-9573-fde4b07cb549 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:37.647730 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a1a1d4-c002-45d6-9573-fde4b07cb549 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:37.675159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-a9a1a1d4-c002-45d6-9573-fde4b07cb549 None None] MFA Rules not processed for user `eb498be561b6468faac6bea1c12edb87`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:41:37.676366 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a9a1a1d4-c002-45d6-9573-fde4b07cb549 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:37.739308 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1431/2879] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:41:37 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:41:37.744716 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3364457e-1c3e-428a-98f7-45854828c3ca admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:37.805233 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3364457e-1c3e-428a-98f7-45854828c3ca admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:37.809854 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4b5fc6-c54f-493e-b90a-c35e392a2102 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:37.810445 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4b5fc6-c54f-493e-b90a-c35e392a2102 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:37.810725 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4b5fc6-c54f-493e-b90a-c35e392a2102 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:37.810997 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4b5fc6-c54f-493e-b90a-c35e392a2102 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:37.811627 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4b5fc6-c54f-493e-b90a-c35e392a2102 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:37.813571 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4b5fc6-c54f-493e-b90a-c35e392a2102 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:37.840142 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1449/2880] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:41:37 2022] POST /identity/v3/projects => generated 363 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:41:37.844258 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a1a1d4-c002-45d6-9573-fde4b07cb549 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:37.855186 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a1a1d4-c002-45d6-9573-fde4b07cb549 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:37.913293 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01705474-d508-4adc-a32b-d31b9bb87885 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:37.913931 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01705474-d508-4adc-a32b-d31b9bb87885 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:37.914225 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01705474-d508-4adc-a32b-d31b9bb87885 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:37.914555 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01705474-d508-4adc-a32b-d31b9bb87885 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:37.915207 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01705474-d508-4adc-a32b-d31b9bb87885 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:37.917228 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01705474-d508-4adc-a32b-d31b9bb87885 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:37.938895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1432/2881] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:41:37 2022] POST /identity/v3/users => generated 288 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:41:37.943565 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4b5fc6-c54f-493e-b90a-c35e392a2102 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:37.954303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4b5fc6-c54f-493e-b90a-c35e392a2102 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:37.958889 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a5474a-2775-448a-bfbd-ce79f600595e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:37.959478 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a5474a-2775-448a-bfbd-ce79f600595e admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:37.959758 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a5474a-2775-448a-bfbd-ce79f600595e admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:37.960031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a5474a-2775-448a-bfbd-ce79f600595e admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:37.960670 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a5474a-2775-448a-bfbd-ce79f600595e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:37.962296 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a5474a-2775-448a-bfbd-ce79f600595e admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:37.972717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1450/2882] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:41:37 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:41:37.977319 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01705474-d508-4adc-a32b-d31b9bb87885 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:37.987265 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01705474-d508-4adc-a32b-d31b9bb87885 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:37.992367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c8eb82-de6f-4beb-9233-a31b7b60ef01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:37.993003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c8eb82-de6f-4beb-9233-a31b7b60ef01 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:37.993323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c8eb82-de6f-4beb-9233-a31b7b60ef01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:37.993600 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c8eb82-de6f-4beb-9233-a31b7b60ef01 admin admin] PATH_INFO: `/v3/projects/20071c17eab64fffbb03d008f6289d61/users/165d3f73140f4772810b87734af684e8/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:38.007952 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c8eb82-de6f-4beb-9233-a31b7b60ef01 admin admin] RBAC: Authorizing `identity:create_grant(project_id=20071c17eab64fffbb03d008f6289d61, user_id=165d3f73140f4772810b87734af684e8, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:38.010641 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c8eb82-de6f-4beb-9233-a31b7b60ef01 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:38.023689 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1433/2883] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:41:37 2022] PUT /identity/v3/projects/20071c17eab64fffbb03d008f6289d61/users/165d3f73140f4772810b87734af684e8/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:41:38.027577 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a5474a-2775-448a-bfbd-ce79f600595e admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:38.038198 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a5474a-2775-448a-bfbd-ce79f600595e admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:38.058207 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:41:38.097171 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649206aa-5de0-47cd-8b91-23c36cd03c6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:38.097787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649206aa-5de0-47cd-8b91-23c36cd03c6d admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:38.098103 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649206aa-5de0-47cd-8b91-23c36cd03c6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:38.098304 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649206aa-5de0-47cd-8b91-23c36cd03c6d admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:38.099029 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649206aa-5de0-47cd-8b91-23c36cd03c6d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:38.101130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649206aa-5de0-47cd-8b91-23c36cd03c6d admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:38.111483 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1451/2884] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:41:38 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:41:38.116226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c8eb82-de6f-4beb-9233-a31b7b60ef01 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:38.122140 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:41:38.127628 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c8eb82-de6f-4beb-9233-a31b7b60ef01 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:38.134144 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbba501-8998-4d5a-8260-776a2a57fdf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:38.134807 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbba501-8998-4d5a-8260-776a2a57fdf9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:38.135091 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbba501-8998-4d5a-8260-776a2a57fdf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:38.135367 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbba501-8998-4d5a-8260-776a2a57fdf9 admin admin] PATH_INFO: `/v3/projects/20071c17eab64fffbb03d008f6289d61/users/165d3f73140f4772810b87734af684e8/roles/d17954e7ce544f0d90a512340232c60e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:38.137363 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbba501-8998-4d5a-8260-776a2a57fdf9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=20071c17eab64fffbb03d008f6289d61, user_id=165d3f73140f4772810b87734af684e8, role_id=d17954e7ce544f0d90a512340232c60e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:38.140002 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbba501-8998-4d5a-8260-776a2a57fdf9 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:38.150226 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1434/2885] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:41:38 2022] PUT /identity/v3/projects/20071c17eab64fffbb03d008f6289d61/users/165d3f73140f4772810b87734af684e8/roles/d17954e7ce544f0d90a512340232c60e => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:41:38.160498 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92507160-bd74-42df-9fd6-875722993fd5 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:38.161006 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92507160-bd74-42df-9fd6-875722993fd5 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:38.161153 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92507160-bd74-42df-9fd6-875722993fd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:38.188757 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-92507160-bd74-42df-9fd6-875722993fd5 None None] MFA Rules not processed for user `165d3f73140f4772810b87734af684e8`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:41:38.190051 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-92507160-bd74-42df-9fd6-875722993fd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:38.257795 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1452/2886] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:41:38 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:41:38.269145 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbba501-8998-4d5a-8260-776a2a57fdf9 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:38.271608 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4cbba501-8998-4d5a-8260-776a2a57fdf9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:38.328372 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbba501-8998-4d5a-8260-776a2a57fdf9 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:38.332011 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9555029d-289d-4cab-9f6c-8a5c4bf5235c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:38.332608 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9555029d-289d-4cab-9f6c-8a5c4bf5235c service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:38.332888 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9555029d-289d-4cab-9f6c-8a5c4bf5235c service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:38.333197 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9555029d-289d-4cab-9f6c-8a5c4bf5235c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:38.342507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9555029d-289d-4cab-9f6c-8a5c4bf5235c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:38.343931 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9555029d-289d-4cab-9f6c-8a5c4bf5235c service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:38.353745 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1435/2887] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:41:38 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:41:38.430394 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92507160-bd74-42df-9fd6-875722993fd5 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:38.440519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92507160-bd74-42df-9fd6-875722993fd5 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:38.444816 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e11a4f-7cb8-4049-9f9a-9609e10d9b9d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:38.445490 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e11a4f-7cb8-4049-9f9a-9609e10d9b9d service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:38.445810 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e11a4f-7cb8-4049-9f9a-9609e10d9b9d service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:38.446084 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e11a4f-7cb8-4049-9f9a-9609e10d9b9d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:38.511421 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e11a4f-7cb8-4049-9f9a-9609e10d9b9d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:38.512916 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e11a4f-7cb8-4049-9f9a-9609e10d9b9d service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:38.522346 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1453/2888] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:41:38 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:41:48.349022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=30,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:41:48.517692 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=80 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:41:48.898394 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9555029d-289d-4cab-9f6c-8a5c4bf5235c service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:48.960692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9555029d-289d-4cab-9f6c-8a5c4bf5235c service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:48.965205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f80e3d1-a023-4d1c-92ec-031a7f9da3ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:48.965832 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f80e3d1-a023-4d1c-92ec-031a7f9da3ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:48.966096 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f80e3d1-a023-4d1c-92ec-031a7f9da3ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:48.966365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f80e3d1-a023-4d1c-92ec-031a7f9da3ce admin admin] PATH_INFO: `/v3/users/eb498be561b6468faac6bea1c12edb87` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:48.967456 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f80e3d1-a023-4d1c-92ec-031a7f9da3ce admin admin] RBAC: Authorizing `identity:delete_user(user_id=eb498be561b6468faac6bea1c12edb87)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:48.969398 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f80e3d1-a023-4d1c-92ec-031a7f9da3ce admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:49.015357 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8f80e3d1-a023-4d1c-92ec-031a7f9da3ce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eb498be561b6468faac6bea1c12edb87'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:49.027159 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8f80e3d1-a023-4d1c-92ec-031a7f9da3ce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eb498be561b6468faac6bea1c12edb87'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:49.033948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8f80e3d1-a023-4d1c-92ec-031a7f9da3ce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eb498be561b6468faac6bea1c12edb87'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:49.041921 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8f80e3d1-a023-4d1c-92ec-031a7f9da3ce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eb498be561b6468faac6bea1c12edb87'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:49.042560 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8f80e3d1-a023-4d1c-92ec-031a7f9da3ce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eb498be561b6468faac6bea1c12edb87'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:49.051684 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1436/2889] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:41:48 2022] DELETE /identity/v3/users/eb498be561b6468faac6bea1c12edb87 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:41:49.056737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9946bc7-0b9d-45a1-9f1a-0126ec6e3a69 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:49.057018 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9946bc7-0b9d-45a1-9f1a-0126ec6e3a69 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:49.057310 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9946bc7-0b9d-45a1-9f1a-0126ec6e3a69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:49.075854 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-c9946bc7-0b9d-45a1-9f1a-0126ec6e3a69 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:41:49.077031 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c9946bc7-0b9d-45a1-9f1a-0126ec6e3a69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:49.135638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1454/2890] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:41:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:41:49.145341 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f80e3d1-a023-4d1c-92ec-031a7f9da3ce admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:49.146760 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8f80e3d1-a023-4d1c-92ec-031a7f9da3ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:49.203486 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f80e3d1-a023-4d1c-92ec-031a7f9da3ce admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:49.207107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f956d5-9eae-4f38-8951-10763c0d888d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:49.207707 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f956d5-9eae-4f38-8951-10763c0d888d service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:49.207986 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f956d5-9eae-4f38-8951-10763c0d888d service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:49.208259 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f956d5-9eae-4f38-8951-10763c0d888d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:49.217618 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f956d5-9eae-4f38-8951-10763c0d888d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:49.219054 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f956d5-9eae-4f38-8951-10763c0d888d service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:49.228861 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1437/2891] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:41:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:41:49.485197 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9946bc7-0b9d-45a1-9f1a-0126ec6e3a69 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:49.486665 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c9946bc7-0b9d-45a1-9f1a-0126ec6e3a69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:49.498202 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9946bc7-0b9d-45a1-9f1a-0126ec6e3a69 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:49.503061 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a958795-5ea9-4fad-bdf3-efccf47c5c78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:49.503682 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a958795-5ea9-4fad-bdf3-efccf47c5c78 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:49.503959 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a958795-5ea9-4fad-bdf3-efccf47c5c78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:49.504230 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a958795-5ea9-4fad-bdf3-efccf47c5c78 admin admin] PATH_INFO: `/v3/projects/0b64913b0b0f4521882b906a3c533c70` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:49.507438 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a958795-5ea9-4fad-bdf3-efccf47c5c78 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b64913b0b0f4521882b906a3c533c70)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:49.507438 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a958795-5ea9-4fad-bdf3-efccf47c5c78 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:49.555221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-3a958795-5ea9-4fad-bdf3-efccf47c5c78 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b64913b0b0f4521882b906a3c533c70', 'request_id': 'req-3a958795-5ea9-4fad-bdf3-efccf47c5c78'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:49.560703 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-3a958795-5ea9-4fad-bdf3-efccf47c5c78 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b64913b0b0f4521882b906a3c533c70', 'request_id': 'req-3a958795-5ea9-4fad-bdf3-efccf47c5c78'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:49.568042 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-3a958795-5ea9-4fad-bdf3-efccf47c5c78 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b64913b0b0f4521882b906a3c533c70', 'request_id': 'req-3a958795-5ea9-4fad-bdf3-efccf47c5c78'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:49.579741 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-3a958795-5ea9-4fad-bdf3-efccf47c5c78 admin admin] The token cache is being invalidate because project 0b64913b0b0f4521882b906a3c533c70 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:41:49.579998 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-3a958795-5ea9-4fad-bdf3-efccf47c5c78 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:49.580636 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-3a958795-5ea9-4fad-bdf3-efccf47c5c78 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:49.582156 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1455/2892] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:41:49 2022] DELETE /identity/v3/projects/0b64913b0b0f4521882b906a3c533c70 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:41:49.586793 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f956d5-9eae-4f38-8951-10763c0d888d service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:49.596334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f956d5-9eae-4f38-8951-10763c0d888d service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:49.614552 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:41:49.653715 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607a6da3-34c5-49b5-8867-35422f224170 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:49.654348 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607a6da3-34c5-49b5-8867-35422f224170 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:49.654657 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607a6da3-34c5-49b5-8867-35422f224170 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:49.654931 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607a6da3-34c5-49b5-8867-35422f224170 admin admin] PATH_INFO: `/v3/users/165d3f73140f4772810b87734af684e8` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:49.656046 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607a6da3-34c5-49b5-8867-35422f224170 admin admin] RBAC: Authorizing `identity:delete_user(user_id=165d3f73140f4772810b87734af684e8)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:49.658009 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607a6da3-34c5-49b5-8867-35422f224170 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:49.703751 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-607a6da3-34c5-49b5-8867-35422f224170 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '165d3f73140f4772810b87734af684e8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:49.716116 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-607a6da3-34c5-49b5-8867-35422f224170 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '165d3f73140f4772810b87734af684e8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:49.722989 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-607a6da3-34c5-49b5-8867-35422f224170 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '165d3f73140f4772810b87734af684e8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:49.729603 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-607a6da3-34c5-49b5-8867-35422f224170 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '165d3f73140f4772810b87734af684e8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:49.730217 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-607a6da3-34c5-49b5-8867-35422f224170 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '165d3f73140f4772810b87734af684e8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:49.736235 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1438/2893] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:41:49 2022] DELETE /identity/v3/users/165d3f73140f4772810b87734af684e8 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:41:50.001430 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a958795-5ea9-4fad-bdf3-efccf47c5c78 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:50.003046 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-3a958795-5ea9-4fad-bdf3-efccf47c5c78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:50.016738 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a958795-5ea9-4fad-bdf3-efccf47c5c78 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:50.074679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de1384f-b238-42b0-8a5d-841acefe8c36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:50.075302 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de1384f-b238-42b0-8a5d-841acefe8c36 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:50.075584 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de1384f-b238-42b0-8a5d-841acefe8c36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:50.075883 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de1384f-b238-42b0-8a5d-841acefe8c36 admin admin] PATH_INFO: `/v3/projects/20071c17eab64fffbb03d008f6289d61` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:50.076990 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de1384f-b238-42b0-8a5d-841acefe8c36 admin admin] RBAC: Authorizing `identity:delete_project(project_id=20071c17eab64fffbb03d008f6289d61)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:50.078924 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de1384f-b238-42b0-8a5d-841acefe8c36 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:50.110414 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:41:50.127566 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-4de1384f-b238-42b0-8a5d-841acefe8c36 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '20071c17eab64fffbb03d008f6289d61', 'request_id': 'req-4de1384f-b238-42b0-8a5d-841acefe8c36'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:50.132958 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-4de1384f-b238-42b0-8a5d-841acefe8c36 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '20071c17eab64fffbb03d008f6289d61', 'request_id': 'req-4de1384f-b238-42b0-8a5d-841acefe8c36'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:50.140169 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-4de1384f-b238-42b0-8a5d-841acefe8c36 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '20071c17eab64fffbb03d008f6289d61', 'request_id': 'req-4de1384f-b238-42b0-8a5d-841acefe8c36'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:50.151783 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-4de1384f-b238-42b0-8a5d-841acefe8c36 admin admin] The token cache is being invalidate because project 20071c17eab64fffbb03d008f6289d61 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:41:50.152099 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-4de1384f-b238-42b0-8a5d-841acefe8c36 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:50.152756 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-4de1384f-b238-42b0-8a5d-841acefe8c36 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:50.154255 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1456/2894] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:41:50 2022] DELETE /identity/v3/projects/20071c17eab64fffbb03d008f6289d61 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:41:50.159034 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d319188e-4994-4e37-8eb3-acae24ffc99a None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:50.159317 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d319188e-4994-4e37-8eb3-acae24ffc99a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:50.159591 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d319188e-4994-4e37-8eb3-acae24ffc99a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:50.179396 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-d319188e-4994-4e37-8eb3-acae24ffc99a None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:41:50.180596 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d319188e-4994-4e37-8eb3-acae24ffc99a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:50.238298 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1439/2895] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:41:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:41:50.247409 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23656bb-c3fb-48dd-bc64-0aa477e6c3dd None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:50.247698 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23656bb-c3fb-48dd-bc64-0aa477e6c3dd None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:50.247976 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23656bb-c3fb-48dd-bc64-0aa477e6c3dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:50.267666 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-c23656bb-c3fb-48dd-bc64-0aa477e6c3dd None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:41:50.268869 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c23656bb-c3fb-48dd-bc64-0aa477e6c3dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:50.277192 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1457/2896] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:41:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:41:50.281325 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d319188e-4994-4e37-8eb3-acae24ffc99a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:50.291660 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d319188e-4994-4e37-8eb3-acae24ffc99a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:50.296168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4ac5b8-2bc6-4f32-81a0-4510ff01e1ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:50.296760 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4ac5b8-2bc6-4f32-81a0-4510ff01e1ef admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:50.297061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4ac5b8-2bc6-4f32-81a0-4510ff01e1ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:50.297350 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4ac5b8-2bc6-4f32-81a0-4510ff01e1ef admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:50.298031 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4ac5b8-2bc6-4f32-81a0-4510ff01e1ef admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:41:50.298330 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4ac5b8-2bc6-4f32-81a0-4510ff01e1ef admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:50.299914 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4ac5b8-2bc6-4f32-81a0-4510ff01e1ef admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:50.311370 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1440/2897] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:41:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:41:50.315435 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23656bb-c3fb-48dd-bc64-0aa477e6c3dd None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:50.325357 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23656bb-c3fb-48dd-bc64-0aa477e6c3dd None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:50.330124 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c60ee9c-1356-4673-b01b-2b1bc6c61b03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:50.330752 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c60ee9c-1356-4673-b01b-2b1bc6c61b03 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:50.331032 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c60ee9c-1356-4673-b01b-2b1bc6c61b03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:50.331305 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c60ee9c-1356-4673-b01b-2b1bc6c61b03 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:50.331941 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c60ee9c-1356-4673-b01b-2b1bc6c61b03 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:50.333873 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c60ee9c-1356-4673-b01b-2b1bc6c61b03 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:50.361342 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1458/2898] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:41:50 2022] POST /identity/v3/projects => generated 375 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:41:50.365349 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4ac5b8-2bc6-4f32-81a0-4510ff01e1ef admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:50.375154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4ac5b8-2bc6-4f32-81a0-4510ff01e1ef admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:50.401901 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:41:50.432582 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42b4d8b-5006-44d4-905e-689844c6dd6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:50.433215 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42b4d8b-5006-44d4-905e-689844c6dd6a admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:50.433497 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42b4d8b-5006-44d4-905e-689844c6dd6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:50.433771 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42b4d8b-5006-44d4-905e-689844c6dd6a admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:50.434402 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42b4d8b-5006-44d4-905e-689844c6dd6a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:50.436318 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42b4d8b-5006-44d4-905e-689844c6dd6a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:50.456542 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1441/2899] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:41:50 2022] POST /identity/v3/users => generated 288 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:41:50.461311 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c60ee9c-1356-4673-b01b-2b1bc6c61b03 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:50.471303 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c60ee9c-1356-4673-b01b-2b1bc6c61b03 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:50.475918 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46eb18ec-fb49-4c2a-8f6f-898dc4c81152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:50.476505 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46eb18ec-fb49-4c2a-8f6f-898dc4c81152 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:50.476781 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46eb18ec-fb49-4c2a-8f6f-898dc4c81152 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:50.477122 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46eb18ec-fb49-4c2a-8f6f-898dc4c81152 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:50.477717 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46eb18ec-fb49-4c2a-8f6f-898dc4c81152 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:50.479316 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46eb18ec-fb49-4c2a-8f6f-898dc4c81152 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:50.489270 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1459/2900] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:41:50 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:41:50.493025 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42b4d8b-5006-44d4-905e-689844c6dd6a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:50.502818 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42b4d8b-5006-44d4-905e-689844c6dd6a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:50.507911 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cad193-0a17-4e95-bc8e-9fa1fe3e66d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:50.508497 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cad193-0a17-4e95-bc8e-9fa1fe3e66d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:50.508773 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cad193-0a17-4e95-bc8e-9fa1fe3e66d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:50.509070 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cad193-0a17-4e95-bc8e-9fa1fe3e66d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:50.509737 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cad193-0a17-4e95-bc8e-9fa1fe3e66d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:50.511340 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cad193-0a17-4e95-bc8e-9fa1fe3e66d7 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:50.521922 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1442/2901] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:41:50 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:41:50.526640 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46eb18ec-fb49-4c2a-8f6f-898dc4c81152 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:50.536635 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46eb18ec-fb49-4c2a-8f6f-898dc4c81152 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:50.541621 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b732c68-62cf-4f1a-b8c5-1a6d90f424a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:50.542253 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b732c68-62cf-4f1a-b8c5-1a6d90f424a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:50.542527 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b732c68-62cf-4f1a-b8c5-1a6d90f424a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:50.542792 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b732c68-62cf-4f1a-b8c5-1a6d90f424a4 admin admin] PATH_INFO: `/v3/projects/2e5d120c22dd4eadac93161b31e13294/users/b7ddd81287ec4a4f98d3e4ff37ce5856/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:50.557489 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b732c68-62cf-4f1a-b8c5-1a6d90f424a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e5d120c22dd4eadac93161b31e13294, user_id=b7ddd81287ec4a4f98d3e4ff37ce5856, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:50.560120 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b732c68-62cf-4f1a-b8c5-1a6d90f424a4 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:50.572603 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1460/2902] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:41:50 2022] PUT /identity/v3/projects/2e5d120c22dd4eadac93161b31e13294/users/b7ddd81287ec4a4f98d3e4ff37ce5856/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:41:50.582725 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a70fdf1-d984-46f5-bf33-d82e36bbde4c None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:50.583008 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a70fdf1-d984-46f5-bf33-d82e36bbde4c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:50.583285 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a70fdf1-d984-46f5-bf33-d82e36bbde4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:50.611024 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-0a70fdf1-d984-46f5-bf33-d82e36bbde4c None None] MFA Rules not processed for user `b7ddd81287ec4a4f98d3e4ff37ce5856`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:41:50.612284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0a70fdf1-d984-46f5-bf33-d82e36bbde4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:50.676052 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1443/2903] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:41:50 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:41:50.687163 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b732c68-62cf-4f1a-b8c5-1a6d90f424a4 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:50.688866 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-4b732c68-62cf-4f1a-b8c5-1a6d90f424a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:50.745000 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b732c68-62cf-4f1a-b8c5-1a6d90f424a4 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:50.748643 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e1b999-a142-4c1d-8352-f12145f95e30 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:50.749269 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e1b999-a142-4c1d-8352-f12145f95e30 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:50.749552 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e1b999-a142-4c1d-8352-f12145f95e30 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:50.749832 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e1b999-a142-4c1d-8352-f12145f95e30 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:50.758024 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e1b999-a142-4c1d-8352-f12145f95e30 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:50.759442 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e1b999-a142-4c1d-8352-f12145f95e30 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:50.768188 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1461/2904] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:41:50 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:41:53.761250 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a70fdf1-d984-46f5-bf33-d82e36bbde4c None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:53.821894 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a70fdf1-d984-46f5-bf33-d82e36bbde4c None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:53.827752 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec00a49e-c525-4f75-93dc-23a975f32802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:53.828513 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec00a49e-c525-4f75-93dc-23a975f32802 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:53.828797 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec00a49e-c525-4f75-93dc-23a975f32802 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:53.829100 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec00a49e-c525-4f75-93dc-23a975f32802 admin admin] PATH_INFO: `/v3/users/b7ddd81287ec4a4f98d3e4ff37ce5856` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:53.830285 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec00a49e-c525-4f75-93dc-23a975f32802 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b7ddd81287ec4a4f98d3e4ff37ce5856)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:53.832253 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec00a49e-c525-4f75-93dc-23a975f32802 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:53.875077 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:41:53.883000 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ec00a49e-c525-4f75-93dc-23a975f32802 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b7ddd81287ec4a4f98d3e4ff37ce5856'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:53.896443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ec00a49e-c525-4f75-93dc-23a975f32802 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b7ddd81287ec4a4f98d3e4ff37ce5856'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:53.903709 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ec00a49e-c525-4f75-93dc-23a975f32802 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b7ddd81287ec4a4f98d3e4ff37ce5856'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:53.911363 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ec00a49e-c525-4f75-93dc-23a975f32802 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b7ddd81287ec4a4f98d3e4ff37ce5856'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:53.912191 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ec00a49e-c525-4f75-93dc-23a975f32802 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b7ddd81287ec4a4f98d3e4ff37ce5856'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:53.918390 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1444/2905] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:41:53 2022] DELETE /identity/v3/users/b7ddd81287ec4a4f98d3e4ff37ce5856 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:41:53.923091 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3c2d8e-31cf-447a-b86e-8b774f62f68a None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:53.923569 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3c2d8e-31cf-447a-b86e-8b774f62f68a None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:53.923767 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3c2d8e-31cf-447a-b86e-8b774f62f68a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:53.943668 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-2d3c2d8e-31cf-447a-b86e-8b774f62f68a None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:41:53.944927 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-2d3c2d8e-31cf-447a-b86e-8b774f62f68a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:53.991548 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:DELETE=6,keystone:SELECT=91,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:41:54.006133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1462/2906] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:41:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:41:54.016332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec00a49e-c525-4f75-93dc-23a975f32802 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:54.017753 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ec00a49e-c525-4f75-93dc-23a975f32802 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:54.076624 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec00a49e-c525-4f75-93dc-23a975f32802 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:54.080741 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f6cd55-62ff-443a-ac41-337d2e60e0b6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:54.081504 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f6cd55-62ff-443a-ac41-337d2e60e0b6 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:54.081829 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f6cd55-62ff-443a-ac41-337d2e60e0b6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:54.082154 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f6cd55-62ff-443a-ac41-337d2e60e0b6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:54.093380 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f6cd55-62ff-443a-ac41-337d2e60e0b6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:54.094860 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f6cd55-62ff-443a-ac41-337d2e60e0b6 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:54.105687 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1445/2907] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:41:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:41:54.381300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3c2d8e-31cf-447a-b86e-8b774f62f68a None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:54.382743 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-2d3c2d8e-31cf-447a-b86e-8b774f62f68a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:54.396631 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3c2d8e-31cf-447a-b86e-8b774f62f68a None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:54.401696 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14236bb-c99c-46a9-9235-65db984b05d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:54.402439 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14236bb-c99c-46a9-9235-65db984b05d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:54.402694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14236bb-c99c-46a9-9235-65db984b05d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:54.402870 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14236bb-c99c-46a9-9235-65db984b05d4 admin admin] PATH_INFO: `/v3/projects/2e5d120c22dd4eadac93161b31e13294` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:54.404032 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e14236bb-c99c-46a9-9235-65db984b05d4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2e5d120c22dd4eadac93161b31e13294)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:54.406016 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e14236bb-c99c-46a9-9235-65db984b05d4 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:54.453337 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e14236bb-c99c-46a9-9235-65db984b05d4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2e5d120c22dd4eadac93161b31e13294', 'request_id': 'req-e14236bb-c99c-46a9-9235-65db984b05d4'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:54.458765 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e14236bb-c99c-46a9-9235-65db984b05d4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2e5d120c22dd4eadac93161b31e13294', 'request_id': 'req-e14236bb-c99c-46a9-9235-65db984b05d4'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:54.466644 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e14236bb-c99c-46a9-9235-65db984b05d4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2e5d120c22dd4eadac93161b31e13294', 'request_id': 'req-e14236bb-c99c-46a9-9235-65db984b05d4'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:54.478279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e14236bb-c99c-46a9-9235-65db984b05d4 admin admin] The token cache is being invalidate because project 2e5d120c22dd4eadac93161b31e13294 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81484) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:41:54.478564 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e14236bb-c99c-46a9-9235-65db984b05d4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:54.479173 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-e14236bb-c99c-46a9-9235-65db984b05d4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:41:54.480638 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1463/2908] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:41:54 2022] DELETE /identity/v3/projects/2e5d120c22dd4eadac93161b31e13294 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:41:54.485852 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94143571-f303-4128-95fd-ad8cc040931b None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:54.486165 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94143571-f303-4128-95fd-ad8cc040931b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:54.486464 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94143571-f303-4128-95fd-ad8cc040931b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:54.506041 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-94143571-f303-4128-95fd-ad8cc040931b None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:41:54.506953 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-94143571-f303-4128-95fd-ad8cc040931b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:54.573268 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1446/2909] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:41:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:41:54.581054 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562f1b75-a58e-42b6-825b-72c22d0e3649 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:54.581345 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562f1b75-a58e-42b6-825b-72c22d0e3649 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:54.581618 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562f1b75-a58e-42b6-825b-72c22d0e3649 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:54.601931 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-562f1b75-a58e-42b6-825b-72c22d0e3649 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:41:54.603745 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-562f1b75-a58e-42b6-825b-72c22d0e3649 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:54.612142 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1464/2910] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:41:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:41:54.616444 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94143571-f303-4128-95fd-ad8cc040931b None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:54.626382 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94143571-f303-4128-95fd-ad8cc040931b None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:54.630933 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891cd123-d6e0-4318-9ca3-7fe90374e31d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:54.631533 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891cd123-d6e0-4318-9ca3-7fe90374e31d admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:54.631814 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891cd123-d6e0-4318-9ca3-7fe90374e31d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:54.632098 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891cd123-d6e0-4318-9ca3-7fe90374e31d admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:54.632744 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891cd123-d6e0-4318-9ca3-7fe90374e31d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:41:54.633061 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891cd123-d6e0-4318-9ca3-7fe90374e31d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:54.634692 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891cd123-d6e0-4318-9ca3-7fe90374e31d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:54.646483 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1447/2911] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:41:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 04:41:54.651545 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562f1b75-a58e-42b6-825b-72c22d0e3649 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:54.660805 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562f1b75-a58e-42b6-825b-72c22d0e3649 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:54.666555 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b09a66-b69c-4442-a4fb-17a530771a53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:54.667192 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b09a66-b69c-4442-a4fb-17a530771a53 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:54.667534 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b09a66-b69c-4442-a4fb-17a530771a53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:54.667949 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b09a66-b69c-4442-a4fb-17a530771a53 admin admin] PATH_INFO: `/v3/projects` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:54.668619 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b09a66-b69c-4442-a4fb-17a530771a53 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:54.670641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b09a66-b69c-4442-a4fb-17a530771a53 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:54.697960 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1465/2912] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:41:54 2022] POST /identity/v3/projects => generated 363 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:41:54.702168 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891cd123-d6e0-4318-9ca3-7fe90374e31d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:54.712504 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891cd123-d6e0-4318-9ca3-7fe90374e31d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:54.757256 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:DELETE=5,keystone:SELECT=94,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:41:54.770257 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2b6534-7ccd-42ca-aa5c-1878ebf478bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:54.770851 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2b6534-7ccd-42ca-aa5c-1878ebf478bf admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:54.771117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2b6534-7ccd-42ca-aa5c-1878ebf478bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:54.771387 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2b6534-7ccd-42ca-aa5c-1878ebf478bf admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:54.772012 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2b6534-7ccd-42ca-aa5c-1878ebf478bf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:54.774465 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2b6534-7ccd-42ca-aa5c-1878ebf478bf admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:54.795175 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1448/2913] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:41:54 2022] POST /identity/v3/users => generated 282 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:41:54.799690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b09a66-b69c-4442-a4fb-17a530771a53 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:54.810390 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b09a66-b69c-4442-a4fb-17a530771a53 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:54.815675 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60799a48-5e34-493c-998b-2cffefdeedd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:54.816302 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60799a48-5e34-493c-998b-2cffefdeedd1 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:54.816632 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60799a48-5e34-493c-998b-2cffefdeedd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:54.816980 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60799a48-5e34-493c-998b-2cffefdeedd1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:54.817787 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60799a48-5e34-493c-998b-2cffefdeedd1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:54.819644 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60799a48-5e34-493c-998b-2cffefdeedd1 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:54.831870 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1466/2914] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:41:54 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:41:54.836250 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2b6534-7ccd-42ca-aa5c-1878ebf478bf admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:54.848880 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2b6534-7ccd-42ca-aa5c-1878ebf478bf admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:54.853939 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f6c3b3-bb70-4aac-8410-06ab80950523 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:54.854535 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f6c3b3-bb70-4aac-8410-06ab80950523 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:54.854809 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f6c3b3-bb70-4aac-8410-06ab80950523 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:54.855077 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f6c3b3-bb70-4aac-8410-06ab80950523 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:54.855714 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f6c3b3-bb70-4aac-8410-06ab80950523 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:54.857870 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f6c3b3-bb70-4aac-8410-06ab80950523 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:54.870857 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1449/2915] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:41:54 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 04:41:54.875178 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60799a48-5e34-493c-998b-2cffefdeedd1 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:54.886516 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60799a48-5e34-493c-998b-2cffefdeedd1 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:54.891258 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a704bd56-65bd-417f-8308-4b3fa75e7e06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:54.891905 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a704bd56-65bd-417f-8308-4b3fa75e7e06 admin admin] REQUEST_METHOD: `PUT` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:54.892185 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a704bd56-65bd-417f-8308-4b3fa75e7e06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:54.892461 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a704bd56-65bd-417f-8308-4b3fa75e7e06 admin admin] PATH_INFO: `/v3/projects/98e6c6ce1921461fa32e1d7ab0e5a967/users/c15c1fc5e60e482d9c8e229805851329/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:54.908110 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a704bd56-65bd-417f-8308-4b3fa75e7e06 admin admin] RBAC: Authorizing `identity:create_grant(project_id=98e6c6ce1921461fa32e1d7ab0e5a967, user_id=c15c1fc5e60e482d9c8e229805851329, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:54.910966 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a704bd56-65bd-417f-8308-4b3fa75e7e06 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:54.924212 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1467/2916] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:41:54 2022] PUT /identity/v3/projects/98e6c6ce1921461fa32e1d7ab0e5a967/users/c15c1fc5e60e482d9c8e229805851329/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:41:54.936113 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3f999a-856b-414c-87fb-23e9c1f9493b None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:54.936399 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3f999a-856b-414c-87fb-23e9c1f9493b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:54.936673 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3f999a-856b-414c-87fb-23e9c1f9493b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:54.964819 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-2e3f999a-856b-414c-87fb-23e9c1f9493b None None] MFA Rules not processed for user `c15c1fc5e60e482d9c8e229805851329`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:41:54.966058 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2e3f999a-856b-414c-87fb-23e9c1f9493b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:55.028661 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1450/2917] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:41:54 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:41:55.039726 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a704bd56-65bd-417f-8308-4b3fa75e7e06 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:41:55.041475 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-a704bd56-65bd-417f-8308-4b3fa75e7e06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:41:55.099548 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a704bd56-65bd-417f-8308-4b3fa75e7e06 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:41:55.103307 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f9fa23-68cf-4540-a8a5-fec1589307d6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:41:55.103910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f9fa23-68cf-4540-a8a5-fec1589307d6 service cinder] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:41:55.104190 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f9fa23-68cf-4540-a8a5-fec1589307d6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:41:55.104463 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f9fa23-68cf-4540-a8a5-fec1589307d6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:41:55.113660 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f9fa23-68cf-4540-a8a5-fec1589307d6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:41:55.115100 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f9fa23-68cf-4540-a8a5-fec1589307d6 service cinder] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:41:55.124262 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1468/2918] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:41:55 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:42:03.907626 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3f999a-856b-414c-87fb-23e9c1f9493b None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:03.909084 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2e3f999a-856b-414c-87fb-23e9c1f9493b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:03.965883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3f999a-856b-414c-87fb-23e9c1f9493b None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:03.969422 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e91bd2-2f8f-44af-9395-a32024a3fad8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:03.970019 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e91bd2-2f8f-44af-9395-a32024a3fad8 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:03.970296 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e91bd2-2f8f-44af-9395-a32024a3fad8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:03.970570 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e91bd2-2f8f-44af-9395-a32024a3fad8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:03.972334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-01e91bd2-2f8f-44af-9395-a32024a3fad8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:04.042381 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:42:04.045343 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e91bd2-2f8f-44af-9395-a32024a3fad8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:04.046954 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e91bd2-2f8f-44af-9395-a32024a3fad8 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:04.065360 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1451/2919] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:42:03 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:42:05.120030 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=83,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:42:05.257231 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f9fa23-68cf-4540-a8a5-fec1589307d6 service cinder] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:05.268185 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f9fa23-68cf-4540-a8a5-fec1589307d6 service cinder] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:05.272067 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f4b9da-4057-4261-92c6-c7be52dbf2d7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:05.272663 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f4b9da-4057-4261-92c6-c7be52dbf2d7 service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:05.272937 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f4b9da-4057-4261-92c6-c7be52dbf2d7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:05.273236 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f4b9da-4057-4261-92c6-c7be52dbf2d7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:05.275679 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-18f4b9da-4057-4261-92c6-c7be52dbf2d7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:05.291355 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f4b9da-4057-4261-92c6-c7be52dbf2d7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:05.292784 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f4b9da-4057-4261-92c6-c7be52dbf2d7 service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:05.311651 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1469/2920] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:42:05 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:42:14.062712 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:42:15.306348 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=6 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:42:15.389848 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e91bd2-2f8f-44af-9395-a32024a3fad8 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:15.391262 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-01e91bd2-2f8f-44af-9395-a32024a3fad8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:15.401906 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e91bd2-2f8f-44af-9395-a32024a3fad8 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:15.406050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bce4b52-016f-4d32-af90-06cd3849150b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:15.406640 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bce4b52-016f-4d32-af90-06cd3849150b service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:15.406916 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bce4b52-016f-4d32-af90-06cd3849150b service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:15.407185 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bce4b52-016f-4d32-af90-06cd3849150b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:15.409021 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2bce4b52-016f-4d32-af90-06cd3849150b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:15.421499 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bce4b52-016f-4d32-af90-06cd3849150b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:15.422929 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bce4b52-016f-4d32-af90-06cd3849150b service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:15.431948 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1452/2921] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:42:15 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:42:16.934324 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f4b9da-4057-4261-92c6-c7be52dbf2d7 service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:16.939254 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-18f4b9da-4057-4261-92c6-c7be52dbf2d7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:16.955139 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f4b9da-4057-4261-92c6-c7be52dbf2d7 service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:16.960889 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408eb19d-acb9-4f43-ae15-16c9db5402d1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc4d455ebf041d488b1199ca58358a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'ResellerAdmin', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:16.961681 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408eb19d-acb9-4f43-ae15-16c9db5402d1 service nova] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:16.962130 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408eb19d-acb9-4f43-ae15-16c9db5402d1 service nova] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:16.966910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408eb19d-acb9-4f43-ae15-16c9db5402d1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:16.966910 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-408eb19d-acb9-4f43-ae15-16c9db5402d1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:16.979102 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408eb19d-acb9-4f43-ae15-16c9db5402d1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:16.980690 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408eb19d-acb9-4f43-ae15-16c9db5402d1 service nova] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:16.992509 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1470/2922] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:42:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:42:20.068405 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bce4b52-016f-4d32-af90-06cd3849150b service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:20.077874 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bce4b52-016f-4d32-af90-06cd3849150b service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:20.081421 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f749525-5061-46bf-a9c7-032d1d5a3cd8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:20.082010 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f749525-5061-46bf-a9c7-032d1d5a3cd8 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:20.082290 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f749525-5061-46bf-a9c7-032d1d5a3cd8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:20.082562 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f749525-5061-46bf-a9c7-032d1d5a3cd8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:20.084282 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6f749525-5061-46bf-a9c7-032d1d5a3cd8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:20.149571 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f749525-5061-46bf-a9c7-032d1d5a3cd8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:20.151034 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f749525-5061-46bf-a9c7-032d1d5a3cd8 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:20.161952 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1453/2923] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:42:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:42:26.284788 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408eb19d-acb9-4f43-ae15-16c9db5402d1 service nova] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:26.286464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-408eb19d-acb9-4f43-ae15-16c9db5402d1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:26.300518 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408eb19d-acb9-4f43-ae15-16c9db5402d1 service nova] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:26.305272 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f36897-4099-47c1-8e06-d3c16d1a8d82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:26.305899 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f36897-4099-47c1-8e06-d3c16d1a8d82 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:26.306180 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f36897-4099-47c1-8e06-d3c16d1a8d82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:26.306574 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f36897-4099-47c1-8e06-d3c16d1a8d82 admin admin] PATH_INFO: `/v3/users/2625939a0b9a43d59452df9ba9982ecb` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:26.307766 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f36897-4099-47c1-8e06-d3c16d1a8d82 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2625939a0b9a43d59452df9ba9982ecb)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:26.309721 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f36897-4099-47c1-8e06-d3c16d1a8d82 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:26.363464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f2f36897-4099-47c1-8e06-d3c16d1a8d82 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2625939a0b9a43d59452df9ba9982ecb'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:26.370951 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f2f36897-4099-47c1-8e06-d3c16d1a8d82 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2625939a0b9a43d59452df9ba9982ecb'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:26.371553 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f2f36897-4099-47c1-8e06-d3c16d1a8d82 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2625939a0b9a43d59452df9ba9982ecb'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:26.384287 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f2f36897-4099-47c1-8e06-d3c16d1a8d82 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2625939a0b9a43d59452df9ba9982ecb'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:26.384920 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-f2f36897-4099-47c1-8e06-d3c16d1a8d82 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2625939a0b9a43d59452df9ba9982ecb'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:26.396934 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1471/2924] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:42:26 2022] DELETE /identity/v3/users/2625939a0b9a43d59452df9ba9982ecb => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:42:26.569904 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f749525-5061-46bf-a9c7-032d1d5a3cd8 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:26.580234 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f749525-5061-46bf-a9c7-032d1d5a3cd8 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:26.646174 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29db54cf-e690-4baf-bdc0-a209a39906e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:26.646818 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29db54cf-e690-4baf-bdc0-a209a39906e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:26.647101 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29db54cf-e690-4baf-bdc0-a209a39906e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:26.647382 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29db54cf-e690-4baf-bdc0-a209a39906e6 admin admin] PATH_INFO: `/v3/projects/18e4bf7297124da4b43ccaeda048f772` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:26.648507 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29db54cf-e690-4baf-bdc0-a209a39906e6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=18e4bf7297124da4b43ccaeda048f772)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:26.650454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29db54cf-e690-4baf-bdc0-a209a39906e6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:26.699690 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-29db54cf-e690-4baf-bdc0-a209a39906e6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '18e4bf7297124da4b43ccaeda048f772', 'request_id': 'req-29db54cf-e690-4baf-bdc0-a209a39906e6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:26.706085 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-29db54cf-e690-4baf-bdc0-a209a39906e6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '18e4bf7297124da4b43ccaeda048f772', 'request_id': 'req-29db54cf-e690-4baf-bdc0-a209a39906e6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:26.714022 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-29db54cf-e690-4baf-bdc0-a209a39906e6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '18e4bf7297124da4b43ccaeda048f772', 'request_id': 'req-29db54cf-e690-4baf-bdc0-a209a39906e6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:26.725667 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-29db54cf-e690-4baf-bdc0-a209a39906e6 admin admin] The token cache is being invalidate because project 18e4bf7297124da4b43ccaeda048f772 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:42:26.725966 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-29db54cf-e690-4baf-bdc0-a209a39906e6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:26.726615 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-29db54cf-e690-4baf-bdc0-a209a39906e6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:26.728108 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1454/2925] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:42:26 2022] DELETE /identity/v3/projects/18e4bf7297124da4b43ccaeda048f772 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:42:26.732314 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f36897-4099-47c1-8e06-d3c16d1a8d82 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:26.734449 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-f2f36897-4099-47c1-8e06-d3c16d1a8d82 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:26.748515 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f36897-4099-47c1-8e06-d3c16d1a8d82 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:26.805799 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d43aa9-cc43-405a-b3e7-5b07f7beacc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:26.806425 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d43aa9-cc43-405a-b3e7-5b07f7beacc3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:26.806699 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d43aa9-cc43-405a-b3e7-5b07f7beacc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:26.806967 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d43aa9-cc43-405a-b3e7-5b07f7beacc3 admin admin] PATH_INFO: `/v3/users/8ce000ee58dc4352a91e77ec52fcb26b` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:26.808127 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d43aa9-cc43-405a-b3e7-5b07f7beacc3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8ce000ee58dc4352a91e77ec52fcb26b)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:26.810098 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d43aa9-cc43-405a-b3e7-5b07f7beacc3 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:26.855251 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b5d43aa9-cc43-405a-b3e7-5b07f7beacc3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8ce000ee58dc4352a91e77ec52fcb26b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:26.863386 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b5d43aa9-cc43-405a-b3e7-5b07f7beacc3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8ce000ee58dc4352a91e77ec52fcb26b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:26.864082 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b5d43aa9-cc43-405a-b3e7-5b07f7beacc3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8ce000ee58dc4352a91e77ec52fcb26b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:26.869472 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:42:26.878178 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b5d43aa9-cc43-405a-b3e7-5b07f7beacc3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8ce000ee58dc4352a91e77ec52fcb26b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:26.879011 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-b5d43aa9-cc43-405a-b3e7-5b07f7beacc3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8ce000ee58dc4352a91e77ec52fcb26b'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:26.891133 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1472/2926] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:42:26 2022] DELETE /identity/v3/users/8ce000ee58dc4352a91e77ec52fcb26b => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:42:27.062112 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29db54cf-e690-4baf-bdc0-a209a39906e6 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:27.063802 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-29db54cf-e690-4baf-bdc0-a209a39906e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:27.078394 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29db54cf-e690-4baf-bdc0-a209a39906e6 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:27.088410 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:42:27.140660 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94baae6d-879d-48d6-85ce-dd44a8ca8588 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:27.141321 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94baae6d-879d-48d6-85ce-dd44a8ca8588 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:27.141605 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94baae6d-879d-48d6-85ce-dd44a8ca8588 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:27.141878 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94baae6d-879d-48d6-85ce-dd44a8ca8588 admin admin] PATH_INFO: `/v3/projects/af87dd63dbe545c3a99753b771eb2cfc` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:27.142973 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94baae6d-879d-48d6-85ce-dd44a8ca8588 admin admin] RBAC: Authorizing `identity:delete_project(project_id=af87dd63dbe545c3a99753b771eb2cfc)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:27.144904 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94baae6d-879d-48d6-85ce-dd44a8ca8588 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:27.189525 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-94baae6d-879d-48d6-85ce-dd44a8ca8588 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'af87dd63dbe545c3a99753b771eb2cfc', 'request_id': 'req-94baae6d-879d-48d6-85ce-dd44a8ca8588'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:27.195015 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-94baae6d-879d-48d6-85ce-dd44a8ca8588 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'af87dd63dbe545c3a99753b771eb2cfc', 'request_id': 'req-94baae6d-879d-48d6-85ce-dd44a8ca8588'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:27.202884 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-94baae6d-879d-48d6-85ce-dd44a8ca8588 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'af87dd63dbe545c3a99753b771eb2cfc', 'request_id': 'req-94baae6d-879d-48d6-85ce-dd44a8ca8588'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:27.214050 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-94baae6d-879d-48d6-85ce-dd44a8ca8588 admin admin] The token cache is being invalidate because project af87dd63dbe545c3a99753b771eb2cfc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:42:27.214332 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-94baae6d-879d-48d6-85ce-dd44a8ca8588 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:27.214956 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-94baae6d-879d-48d6-85ce-dd44a8ca8588 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:27.216478 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1455/2927] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:42:27 2022] DELETE /identity/v3/projects/af87dd63dbe545c3a99753b771eb2cfc => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:42:33.446040 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d43aa9-cc43-405a-b3e7-5b07f7beacc3 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:33.447464 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-b5d43aa9-cc43-405a-b3e7-5b07f7beacc3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:33.460636 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d43aa9-cc43-405a-b3e7-5b07f7beacc3 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:33.516431 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653cc75c-c525-4691-a142-1562e383511b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:33.517085 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653cc75c-c525-4691-a142-1562e383511b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:33.517369 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653cc75c-c525-4691-a142-1562e383511b admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:33.517641 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653cc75c-c525-4691-a142-1562e383511b admin admin] PATH_INFO: `/v3/users/c15c1fc5e60e482d9c8e229805851329` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:33.518759 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653cc75c-c525-4691-a142-1562e383511b admin admin] RBAC: Authorizing `identity:delete_user(user_id=c15c1fc5e60e482d9c8e229805851329)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:33.520663 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653cc75c-c525-4691-a142-1562e383511b admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:33.568982 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-653cc75c-c525-4691-a142-1562e383511b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c15c1fc5e60e482d9c8e229805851329'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:33.581972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-653cc75c-c525-4691-a142-1562e383511b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c15c1fc5e60e482d9c8e229805851329'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:33.582861 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-653cc75c-c525-4691-a142-1562e383511b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c15c1fc5e60e482d9c8e229805851329'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:33.595339 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-653cc75c-c525-4691-a142-1562e383511b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c15c1fc5e60e482d9c8e229805851329'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:33.595991 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-653cc75c-c525-4691-a142-1562e383511b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c15c1fc5e60e482d9c8e229805851329'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:33.609143 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1473/2928] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:42:33 2022] DELETE /identity/v3/users/c15c1fc5e60e482d9c8e229805851329 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:42:33.613673 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7176c1-8a8b-4f7b-89d2-ad8b8002bdd1 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:33.614446 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7176c1-8a8b-4f7b-89d2-ad8b8002bdd1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:33.614625 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7176c1-8a8b-4f7b-89d2-ad8b8002bdd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:33.634912 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-db7176c1-8a8b-4f7b-89d2-ad8b8002bdd1 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:42:33.636107 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-db7176c1-8a8b-4f7b-89d2-ad8b8002bdd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:33.696563 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1456/2929] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:42:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:42:33.706894 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653cc75c-c525-4691-a142-1562e383511b admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:33.708331 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-653cc75c-c525-4691-a142-1562e383511b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:33.765281 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653cc75c-c525-4691-a142-1562e383511b admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:33.768984 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957171f9-b32a-44cc-bfb5-91dd935d14ff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:33.769636 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957171f9-b32a-44cc-bfb5-91dd935d14ff service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:33.769915 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957171f9-b32a-44cc-bfb5-91dd935d14ff service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:33.770181 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957171f9-b32a-44cc-bfb5-91dd935d14ff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:33.780004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957171f9-b32a-44cc-bfb5-91dd935d14ff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:33.781468 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957171f9-b32a-44cc-bfb5-91dd935d14ff service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:33.786366 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:42:33.792600 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1474/2930] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:42:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 04:42:34.058232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7176c1-8a8b-4f7b-89d2-ad8b8002bdd1 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:34.059907 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-db7176c1-8a8b-4f7b-89d2-ad8b8002bdd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:34.074680 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7176c1-8a8b-4f7b-89d2-ad8b8002bdd1 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:34.080581 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0f5174-29d0-4a59-8312-a3ed5185a6ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:34.081259 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0f5174-29d0-4a59-8312-a3ed5185a6ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:34.081537 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0f5174-29d0-4a59-8312-a3ed5185a6ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:34.081807 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0f5174-29d0-4a59-8312-a3ed5185a6ea admin admin] PATH_INFO: `/v3/projects/98e6c6ce1921461fa32e1d7ab0e5a967` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:34.084041 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0f5174-29d0-4a59-8312-a3ed5185a6ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=98e6c6ce1921461fa32e1d7ab0e5a967)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:34.086000 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0f5174-29d0-4a59-8312-a3ed5185a6ea admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:34.128277 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:42:34.139309 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2c0f5174-29d0-4a59-8312-a3ed5185a6ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '98e6c6ce1921461fa32e1d7ab0e5a967', 'request_id': 'req-2c0f5174-29d0-4a59-8312-a3ed5185a6ea'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:34.145116 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2c0f5174-29d0-4a59-8312-a3ed5185a6ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '98e6c6ce1921461fa32e1d7ab0e5a967', 'request_id': 'req-2c0f5174-29d0-4a59-8312-a3ed5185a6ea'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:34.152365 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2c0f5174-29d0-4a59-8312-a3ed5185a6ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '98e6c6ce1921461fa32e1d7ab0e5a967', 'request_id': 'req-2c0f5174-29d0-4a59-8312-a3ed5185a6ea'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:34.164236 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2c0f5174-29d0-4a59-8312-a3ed5185a6ea admin admin] The token cache is being invalidate because project 98e6c6ce1921461fa32e1d7ab0e5a967 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:42:34.164525 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2c0f5174-29d0-4a59-8312-a3ed5185a6ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:34.165205 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2c0f5174-29d0-4a59-8312-a3ed5185a6ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:34.166688 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1457/2931] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:42:34 2022] DELETE /identity/v3/projects/98e6c6ce1921461fa32e1d7ab0e5a967 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:42:34.171646 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a88fdd-5c4d-470b-9e75-c4dc48f4df40 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:34.171970 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a88fdd-5c4d-470b-9e75-c4dc48f4df40 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:34.172343 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a88fdd-5c4d-470b-9e75-c4dc48f4df40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:34.193842 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-d1a88fdd-5c4d-470b-9e75-c4dc48f4df40 None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:42:34.195111 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d1a88fdd-5c4d-470b-9e75-c4dc48f4df40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:34.259430 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1475/2932] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:42:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:42:34.267791 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507ec8fc-6fa4-465a-abde-51959020d19d None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:34.268074 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507ec8fc-6fa4-465a-abde-51959020d19d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:34.268345 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507ec8fc-6fa4-465a-abde-51959020d19d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:34.287994 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-507ec8fc-6fa4-465a-abde-51959020d19d None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:42:34.289251 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-507ec8fc-6fa4-465a-abde-51959020d19d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:34.297232 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1458/2933] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:42:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:42:34.301621 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a88fdd-5c4d-470b-9e75-c4dc48f4df40 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:34.311854 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a88fdd-5c4d-470b-9e75-c4dc48f4df40 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:34.316412 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289a7fa5-26d5-4b4d-bab1-fd5cb3f0bc4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:34.317012 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289a7fa5-26d5-4b4d-bab1-fd5cb3f0bc4d admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:34.317319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289a7fa5-26d5-4b4d-bab1-fd5cb3f0bc4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:34.317594 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289a7fa5-26d5-4b4d-bab1-fd5cb3f0bc4d admin admin] PATH_INFO: `/v3/domains` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:34.318293 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289a7fa5-26d5-4b4d-bab1-fd5cb3f0bc4d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81484) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 04:42:34.318589 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289a7fa5-26d5-4b4d-bab1-fd5cb3f0bc4d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:34.320181 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289a7fa5-26d5-4b4d-bab1-fd5cb3f0bc4d admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:34.332709 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1476/2934] 10.222.0.85 () {58 vars in 1153 bytes} [Tue Aug 30 04:42:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 04:42:34.337151 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507ec8fc-6fa4-465a-abde-51959020d19d None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:34.346316 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507ec8fc-6fa4-465a-abde-51959020d19d None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:34.350958 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab3cf10-2dea-40b1-be94-cc1a8aa018fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:34.351543 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab3cf10-2dea-40b1-be94-cc1a8aa018fc admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:34.351818 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab3cf10-2dea-40b1-be94-cc1a8aa018fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:34.352081 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab3cf10-2dea-40b1-be94-cc1a8aa018fc admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:34.352704 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab3cf10-2dea-40b1-be94-cc1a8aa018fc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:34.354634 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab3cf10-2dea-40b1-be94-cc1a8aa018fc admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:34.383003 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1459/2935] 10.222.0.85 () {60 vars in 1153 bytes} [Tue Aug 30 04:42:34 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 04:42:34.387197 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289a7fa5-26d5-4b4d-bab1-fd5cb3f0bc4d admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:34.397804 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289a7fa5-26d5-4b4d-bab1-fd5cb3f0bc4d admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:34.455446 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbb247f-e59d-444f-8441-8450bb9922a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:34.456189 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbb247f-e59d-444f-8441-8450bb9922a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:34.456319 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbb247f-e59d-444f-8441-8450bb9922a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:34.456589 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbb247f-e59d-444f-8441-8450bb9922a8 admin admin] PATH_INFO: `/v3/users` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:34.457262 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fbb247f-e59d-444f-8441-8450bb9922a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:34.459185 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fbb247f-e59d-444f-8441-8450bb9922a8 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:34.478990 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1477/2936] 10.222.0.85 () {60 vars in 1144 bytes} [Tue Aug 30 04:42:34 2022] POST /identity/v3/users => generated 287 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 04:42:34.483547 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab3cf10-2dea-40b1-be94-cc1a8aa018fc admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:34.495018 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab3cf10-2dea-40b1-be94-cc1a8aa018fc admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:34.500284 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2518adb8-230c-4a5c-88ce-466acbfb8473 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:34.500872 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2518adb8-230c-4a5c-88ce-466acbfb8473 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:34.501188 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2518adb8-230c-4a5c-88ce-466acbfb8473 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:34.501463 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2518adb8-230c-4a5c-88ce-466acbfb8473 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:34.502106 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2518adb8-230c-4a5c-88ce-466acbfb8473 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:34.503706 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2518adb8-230c-4a5c-88ce-466acbfb8473 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:34.515384 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1460/2937] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:42:34 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:42:34.519359 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbb247f-e59d-444f-8441-8450bb9922a8 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:34.529257 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbb247f-e59d-444f-8441-8450bb9922a8 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:34.533951 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd37394-1b90-48aa-b347-62f41660fbc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:34.534542 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd37394-1b90-48aa-b347-62f41660fbc1 admin admin] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:34.534818 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd37394-1b90-48aa-b347-62f41660fbc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:34.535087 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd37394-1b90-48aa-b347-62f41660fbc1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:34.535724 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd37394-1b90-48aa-b347-62f41660fbc1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:34.537333 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd37394-1b90-48aa-b347-62f41660fbc1 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:34.549410 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1478/2938] 10.222.0.85 () {58 vars in 1122 bytes} [Tue Aug 30 04:42:34 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 04:42:34.553553 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2518adb8-230c-4a5c-88ce-466acbfb8473 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:34.564117 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2518adb8-230c-4a5c-88ce-466acbfb8473 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:34.568855 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ccea04-d06a-4049-afd1-84f9ca6d8d07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:34.569594 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ccea04-d06a-4049-afd1-84f9ca6d8d07 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:34.569871 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ccea04-d06a-4049-afd1-84f9ca6d8d07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:34.570139 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ccea04-d06a-4049-afd1-84f9ca6d8d07 admin admin] PATH_INFO: `/v3/projects/148aceebff564c6fa34b4aa8673d45d6/users/6dda792543bb4a99b53842a8fd9fd4c7/roles/ae0cef8120cf4dfc899c30803243a851` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:34.585504 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ccea04-d06a-4049-afd1-84f9ca6d8d07 admin admin] RBAC: Authorizing `identity:create_grant(project_id=148aceebff564c6fa34b4aa8673d45d6, user_id=6dda792543bb4a99b53842a8fd9fd4c7, role_id=ae0cef8120cf4dfc899c30803243a851)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:34.588227 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ccea04-d06a-4049-afd1-84f9ca6d8d07 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:34.601534 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1461/2939] 10.222.0.85 () {60 vars in 1483 bytes} [Tue Aug 30 04:42:34 2022] PUT /identity/v3/projects/148aceebff564c6fa34b4aa8673d45d6/users/6dda792543bb4a99b53842a8fd9fd4c7/roles/ae0cef8120cf4dfc899c30803243a851 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:42:34.613737 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c57ad4e-4779-4a9c-aa26-37f7edf086c8 None None] REQUEST_METHOD: `POST` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:34.614023 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c57ad4e-4779-4a9c-aa26-37f7edf086c8 None None] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:34.614323 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c57ad4e-4779-4a9c-aa26-37f7edf086c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:34.642074 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.auth.core [None req-7c57ad4e-4779-4a9c-aa26-37f7edf086c8 None None] MFA Rules not processed for user `6dda792543bb4a99b53842a8fd9fd4c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81484) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:42:34.643276 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-7c57ad4e-4779-4a9c-aa26-37f7edf086c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:34.700519 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:42:34.710299 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1479/2940] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:42:34 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 04:42:34.721665 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ccea04-d06a-4049-afd1-84f9ca6d8d07 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:34.723403 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f8ccea04-d06a-4049-afd1-84f9ca6d8d07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:34.782102 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ccea04-d06a-4049-afd1-84f9ca6d8d07 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:34.785698 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325e683a-2196-4055-b871-f0bbee2b41a0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7ef280ce39b14085be329f60b22530d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:34.786298 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325e683a-2196-4055-b871-f0bbee2b41a0 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:34.786575 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325e683a-2196-4055-b871-f0bbee2b41a0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:34.786842 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325e683a-2196-4055-b871-f0bbee2b41a0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:34.795943 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-325e683a-2196-4055-b871-f0bbee2b41a0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:34.797654 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-325e683a-2196-4055-b871-f0bbee2b41a0 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:34.806678 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1462/2941] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:42:34 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:42:34.833541 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c57ad4e-4779-4a9c-aa26-37f7edf086c8 None None] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:34.895476 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c57ad4e-4779-4a9c-aa26-37f7edf086c8 None None] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:34.900074 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39f59a8-e1a9-489a-b0e8-2e6ea3cac992 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:34.900694 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39f59a8-e1a9-489a-b0e8-2e6ea3cac992 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:34.900972 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39f59a8-e1a9-489a-b0e8-2e6ea3cac992 admin admin] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:34.901279 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39f59a8-e1a9-489a-b0e8-2e6ea3cac992 admin admin] PATH_INFO: `/v3/users/6dda792543bb4a99b53842a8fd9fd4c7` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:34.902403 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39f59a8-e1a9-489a-b0e8-2e6ea3cac992 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6dda792543bb4a99b53842a8fd9fd4c7)` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:34.904340 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39f59a8-e1a9-489a-b0e8-2e6ea3cac992 admin admin] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:34.951385 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d39f59a8-e1a9-489a-b0e8-2e6ea3cac992 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6dda792543bb4a99b53842a8fd9fd4c7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:34.959004 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d39f59a8-e1a9-489a-b0e8-2e6ea3cac992 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6dda792543bb4a99b53842a8fd9fd4c7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:34.959789 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d39f59a8-e1a9-489a-b0e8-2e6ea3cac992 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6dda792543bb4a99b53842a8fd9fd4c7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:34.972775 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d39f59a8-e1a9-489a-b0e8-2e6ea3cac992 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6dda792543bb4a99b53842a8fd9fd4c7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:34.973509 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.notifications [None req-d39f59a8-e1a9-489a-b0e8-2e6ea3cac992 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6dda792543bb4a99b53842a8fd9fd4c7'} {{(pid=81484) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:34.986430 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1480/2942] 10.222.0.85 () {58 vars in 1224 bytes} [Tue Aug 30 04:42:34 2022] DELETE /identity/v3/users/6dda792543bb4a99b53842a8fd9fd4c7 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 04:42:34.991238 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f18b5ea-5af8-4ebf-a5a4-b0720823a70f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:34.991520 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f18b5ea-5af8-4ebf-a5a4-b0720823a70f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:34.991865 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f18b5ea-5af8-4ebf-a5a4-b0720823a70f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:35.010693 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5f18b5ea-5af8-4ebf-a5a4-b0720823a70f None None] MFA Rules not processed for user `a85a3472aa4a4c95a2bc16b6450e8dcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 04:42:35.011883 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5f18b5ea-5af8-4ebf-a5a4-b0720823a70f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:35.070384 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1463/2943] 10.222.0.85 () {58 vars in 958 bytes} [Tue Aug 30 04:42:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 04:42:35.081221 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39f59a8-e1a9-489a-b0e8-2e6ea3cac992 admin admin] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:35.082721 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-d39f59a8-e1a9-489a-b0e8-2e6ea3cac992 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:35.141232 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39f59a8-e1a9-489a-b0e8-2e6ea3cac992 admin admin] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:35.144928 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94da8396-a5ba-4d38-ae27-83d8f2b4cb9a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5c2edd77896a4534888d6758ec0ee0bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:35.145609 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da8396-a5ba-4d38-ae27-83d8f2b4cb9a service neutron] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:35.145885 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da8396-a5ba-4d38-ae27-83d8f2b4cb9a service neutron] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:35.146154 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da8396-a5ba-4d38-ae27-83d8f2b4cb9a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:35.155671 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94da8396-a5ba-4d38-ae27-83d8f2b4cb9a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:35.157229 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94da8396-a5ba-4d38-ae27-83d8f2b4cb9a service neutron] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:35.166711 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1481/2944] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:42:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:42:35.445989 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f18b5ea-5af8-4ebf-a5a4-b0720823a70f None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:42:35.447454 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5f18b5ea-5af8-4ebf-a5a4-b0720823a70f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:42:35.456448 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:42:35.460787 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f18b5ea-5af8-4ebf-a5a4-b0720823a70f None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:42:35.467443 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309fe276-82bf-42ad-860a-14fbb9e35cc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a85a3472aa4a4c95a2bc16b6450e8dcd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'be2e356bbe7d4680bd52a62e89c77545', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:42:35.468243 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309fe276-82bf-42ad-860a-14fbb9e35cc1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:42:35.468544 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309fe276-82bf-42ad-860a-14fbb9e35cc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:42:35.469118 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309fe276-82bf-42ad-860a-14fbb9e35cc1 admin admin] PATH_INFO: `/v3/projects/148aceebff564c6fa34b4aa8673d45d6` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:42:35.477361 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309fe276-82bf-42ad-860a-14fbb9e35cc1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=148aceebff564c6fa34b4aa8673d45d6)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:42:35.479334 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309fe276-82bf-42ad-860a-14fbb9e35cc1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:42:35.526697 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-309fe276-82bf-42ad-860a-14fbb9e35cc1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '148aceebff564c6fa34b4aa8673d45d6', 'request_id': 'req-309fe276-82bf-42ad-860a-14fbb9e35cc1'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:35.533323 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-309fe276-82bf-42ad-860a-14fbb9e35cc1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '148aceebff564c6fa34b4aa8673d45d6', 'request_id': 'req-309fe276-82bf-42ad-860a-14fbb9e35cc1'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:35.540758 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-309fe276-82bf-42ad-860a-14fbb9e35cc1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '148aceebff564c6fa34b4aa8673d45d6', 'request_id': 'req-309fe276-82bf-42ad-860a-14fbb9e35cc1'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:35.552040 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-309fe276-82bf-42ad-860a-14fbb9e35cc1 admin admin] The token cache is being invalidate because project 148aceebff564c6fa34b4aa8673d45d6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 04:42:35.552333 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-309fe276-82bf-42ad-860a-14fbb9e35cc1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:35.552981 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-309fe276-82bf-42ad-860a-14fbb9e35cc1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 04:42:35.554504 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1464/2945] 10.222.0.85 () {58 vars in 1233 bytes} [Tue Aug 30 04:42:35 2022] DELETE /identity/v3/projects/148aceebff564c6fa34b4aa8673d45d6 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 04:42:45.162433 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=81,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:42:45.548895 dv-cinder-854265-2-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=25,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 04:44:25.725883 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94da8396-a5ba-4d38-ae27-83d8f2b4cb9a service neutron] Authenticating user token {{(pid=81484) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 04:44:25.727300 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-94da8396-a5ba-4d38-ae27-83d8f2b4cb9a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:44:25.874963 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94da8396-a5ba-4d38-ae27-83d8f2b4cb9a service neutron] Validating token access rules against request {{(pid=81484) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 04:44:25.933219 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55e81e3-267c-4c02-8d19-ae60a8aa457d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '359d79cbfcad49beba2b495066824128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a860dcc3b2d64d1491c40710ec7423a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81484) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 04:44:25.933833 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55e81e3-267c-4c02-8d19-ae60a8aa457d service placement] REQUEST_METHOD: `GET` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 04:44:25.934112 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55e81e3-267c-4c02-8d19-ae60a8aa457d service placement] SCRIPT_NAME: `/identity` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 04:44:25.934380 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55e81e3-267c-4c02-8d19-ae60a8aa457d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81484) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 04:44:25.936774 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.fernet_utils [None req-c55e81e3-267c-4c02-8d19-ae60a8aa457d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81484) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 04:44:25.948535 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55e81e3-267c-4c02-8d19-ae60a8aa457d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 04:44:25.950220 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55e81e3-267c-4c02-8d19-ae60a8aa457d service placement] RBAC: Authorization granted {{(pid=81484) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 04:44:25.960354 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: [pid: 81484|app: 0|req: 1482/2946] 10.222.0.85 () {60 vars in 1365 bytes} [Tue Aug 30 04:44:25 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 04:44:35.956087 dv-cinder-854265-2-iscsi devstack@keystone.service[81484]: DEBUG dbcounter [-] [81484] Writing DB stats keystone:SELECT=89 {{(pid=81484) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}